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Abstract 

In  previous  research,  we  documented  our  evolving  research  that  expanded  on  and  formalized  an 
approach  to  the  design  of  a  Performance  Evaluation  (PE)  methodology  for  Data  Fusion  (DF)-based 
tactical  aircraft  systems.  We  have  shown  that  the  design  of  a  PE  process  for  any  multi-sensor  or  multi- 
aircraft  fusion-based  system  involves  the  design  of  a  separate  data  fusion  process  involving  association 
and  estimation  functions  for  PE  purposes  per  se.  Our  publications  to  date  have  developed  the  theoretical 
and  architectural  groundings  for  this  new'  PE  process,  and  several  case  studies  have  been  carried  out  to 
show  sample  implementations  of  the  principles  of  this  new  methodology.  In  addition,  some  limited- 
objective  parametric  experiments  have  also  been  carried  out  that  show  the  application  of  the  new 
evaluation  methodology  for  typical  tactical  aircraft  problems.  In  this  report,  we  summarize  the  findings  of 
these  past  works,  and  show  our  research  efforts  related  to  extending  the  design  and  application  of  this 
methodology  to  air-to-air  engagement  problems  involving  higher-levels  of  data  fusion  capability 
(situation  and  threat  estimation)  and  the  employment  of  electronic  warfare  systems.  The  report  discusses 
the  detailed  strategies  for  data  association,  metrics  estimation,  and  also  the  analytical  techniques  that 
exploit  the  formality  of  the  methods  of  Statistical  Design  of  Experiments  (DOE)  and  Analysis  of  Variance 
(ANOVA)  for  these  fusion  applications.  At  the  end  we  survey  and  study  the  various  methods  available  in 
literature  to  solve  the  large  factor  Design  of  Experiment  problem,  with  a  detailed  guidelines  for 
classification  and  selection  of  a  proper  design. 


CUBRC  |  Introduction 


1  Introduction 


Data  fusion  is  an  information  process  involving  functional  sub-processes  that  align  or  normalize  the 
data  from  several  input  sources  (typically  sensor  data  from  surveillance  and  reconnaissance  sensors  in 
defense  applications),  associate  these  data  to  hypothesized  specific  entities  or  events  or  behaviors  in  the 
observation  space,  and  then  employ  these  associated  or  assigned  observations  toward  developing 
improved  state  estimates  regarding  those  entities  or  events  or  behaviors.  Data  Fusion  (DF  henceforth)  is  a 
relatively  young  technology,  having  had  its  start  in  the  1970’s  driven  largely  by  the  need  to  manage  large 
sensor  data  volumes  from  surveillance  operations  during  the  Cold  War;  DF  was  thus  notionalized  as  a 
kind  of  “data  compression”  technique  in  the  formative  years  of  its  development.  Later,  since  many  DF 
applications  involved  developing  these  fused  state  estimates  for  human  users  of  various  type,  a  more 
holistic  and  systemic  view  of  the  DF  process  was  developed  and  the  overall  process  was  better  defined. 
The  process  can  be  described  as  follows  (Figure  1): 


Figure  1.  Notional  Data  Fusion  Process. 


In  the  typical  defense  application  context,  the  state  of  the  Real  World  is  not  known  but  can  be 
estimated  a  prion,  in  the  usual  approach  to  the  design  of  a  deductivcly-based  or  model-based  approach  to 
the  design  of  an  estimation  or  inferencing  process.  This  dynamic  changing  Real  World  is  observed  by 
multiple  types  of  sensor  systems  as  noted,  and  these  observations  then  need  to  be  set  into  a  common 
frame  of  reference  to  include  e.g.,  transforming  all  observations  to  a  common  coordinate  system,  common 
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time  base,  etc.  Then  the  observation  data  set  needs  to  be  related  to  the  entities  and  features  in  the  model 
knowledge  base;  this  is  done  via  a  Data  Association  process  that  employs  metrics  reflecting  the 
“closeness”  of  any  given  observation  to  a  modeled  entity;  this  process  culminates  in  the  assignment  of  all 
observations  to  the  various  hypothesized  entities.  The  assigned  observations  arc  allocated  to  specific  state 
estimation  algorithms  that  are  specially  designed  to  exploit  the  overall  information  content  of  the  assigned 
observations  to  generate  improved  entity  state  estimates.  The  usual  notion  of  developing  “improved” 
estimates  is  in  the  sense  of  improvements  in  accuracy  but  also  in  the  reduction  of  uncertainty,  as  all  of  the 
input  can  sensibly  be  treated  as  random  variables,  resulting  from  noisy  observations  from  imperfect 
sensor  devices.  These  estimates,  in  many  but  not  all  applications,  arc  often  employed  by  a  human 
user/operator  to  effect  decisions  and  action-taking,  which  themselves  can  possibly  change  the  Real  World 
state  as  noted.  Ideally,  the  DF  process  is  designed  as  an  adaptive  feedback  process,  involving  for  example 
adaptive,  real-time  control  of  the  sensors  (“sensor  management”),  or  dynamic  adaptations  to  the 
algorithmic  processes  (“process  refinement”)  as  shown  above  in  the  notional  feedback  depicted. 
Evaluation  of  this  process  is  approached  w  ithin  the  framework  of  comparing  estimated  states  of  the  Real 
World  with  the  “truth”  states  of  the  Real  World;  truth  states  are  often  only  known  during  the  Test  and 
Evaluation  (T&E)  phase  of  the  design  and  development  of  the  prototype  DF  process,  c.g.,  during 
simulation-based  or  range-testing  of  the  DF  process. 

A  very  typical  application  of  the  DF  process  is  for  the  case  of  multisensor-multitarget  tracking,  i.e.,  the 
case  where  multiple  sensors  are  employed  to  develop  data  that  allows  DF-based  kinematic  state  estimates 
(position,  velocity,  etc)  of  objects  of  interest  (“targets”)  to  be  developed.  This  is  called  “Level  1”  DF, 
wherein  state  estimates  on  single  objects  are  developed;  often  these  LI  estimates  also  include  identity 
estimation  by  fusing  the  observed  entity/feature  data  to  estimate  the  object  class  or  specific  identity  type 
(e.g.,  fighter  aircraft  or  alternately  F-l 6),  depending  on  the  specificity  needed  in  any  given  application. 
While  these  types  of  estimates  are  very  useful  for  military  applications,  as  they  aid  a  commander  in 
assessing  “where  is  it?”  and  “what  is  it?”  more  can  be  done  using  additional  DF  methods.  The  typical 
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next  levels  of  processing,  called  “Level  2:  Situation  estimation”,  and  “Level  3:  Threat  estimation”,  then 
leverage  from  the  Level  estimates  essentially  by  putting  the  LI  estimates  in  context,  i.e.  exploit  contextual 
information  to  develop  the  situational  and  threat  state  estimates. 

2  Developing  an  Approach  to  T&E  for  Data  Fusion  Processes 

2.1  Addressing  the  T&E  Process  for  Level  1  Data  Fusion-based  Tracking  Systems 

The  design  and  development  of  algorithmic  techniques  for  estimating  the  “best”  location  and  related 
kinematic  parameters  of  moving  objects  which  are  observed  by  single  or  multiple  sensors  is  a  complex 
process.  It  is  complicated  in  part  by  the  difficulty  of  obtaining  high-quality  measurements  from  sensor 
systems  due  to  underlying  sensor  limitations  regarding  precision  and  accuracy,  reliability,  etc.,  from 
natural  phenomena  that  complicate  the  observing  process  (weather  effects,  terrain  clutter,  etc),  and  in  the 
defense-problems  of  interest,  from  the  possible  use  of  sensor  countermeasures  employed  (covertly)  by  an 
adversary.  Another  complicating  factor  is  the  inaccuracy  associated  with  the  estimation  algorithm  being 
used.  Virtually  all  estimation  algorithms  are  model-based,  and  employ  a  priori  models  of  target  motion, 
sensor  errors,  system  noises  etc  in  order  to  estimate  the  target  kinematics.  The  process  is  further 
complicated  in  environments  consisting  of  multiple  closely  spaced  targets.  As  a  result,  there  will  be 
differences  between  the  estimated  (from  the  “System  Under  Test  or  SUT”)  and  the  real  (“Truth”)  picture 
of  the  composite  multi-object  kinematic  behavior.  The  goal  of  a  Multi  Target  Tracking  System  (MTTS) 
designer  is  to  develop  a  fusion-based  tracking  system  that  yields  a  composite,  estimated  kinematic  picture 
which  is  in  some  sense  considered  a  “good  enough”  estimate  of  the  composite,  true  object  behavior. 
Hence  at  various  stages  in  development  of  a  tracking  system  it  is  necessary  to  evaluate  the  performance  of 
the  system  in  order  to  see  how  close  the  system’s  estimate  is  to  the  true  picture.  This  is  the  fundamental 
issue  addressed  here:  given  all  the  components  of  a  typical  tracking  system  (whose  design,  as  a  network 
of  separate  fusion  processing  nodes,  is  often  referred  to  as  a  “Data  Fusion  Tree”),  along  with  the 
overarching  stochastic  characteristics  of  the  problem,  on  what  basis  can  an  equitable  approach  to 
evaluation  of  a  candidate-design  tracking  system — the  “SUT”  —  be  based?  This  issue  is  far  ranging  in  that 
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it  applies  to  most  multi-target,  multi-sensor  based  tracking  systems,  although  strictly  speaking  it  only 
applies  when  there  is  ambiguity  in  the  Data  Association  process.  However,  since  we  lack  the  knowledge 
to  quantify  the  degree  to  which  Data  Association  ambiguity  affects  the  need  to  carefully  determine  an 
evaluation  approach,  the  concern  about  this  issue  extends  across  a  broad  range  of  tracking  applications. 

During  the  process  of  designing  and  developing  the  SUT  DF  prototype  process,  the  T&E  phases 
evolve  from  concept  validation  testing  to  developmental  testing;  in  these  phases,  and  often  in  later  in 
controlled  operational  testing,  the  truth  states  of  the  “Real  World”  are  known.  Presuming  the  evaluation 
philosophy  is  based  on  comparisons  between  SUT  DF  process-generated  state  estimates  and  the  truth 
states,  the  known  truth  conditions  usually  allow  for  straightforward  calculation  of  the  various  evaluation 
metrics  being  employed.  Such  evaluation  techniques  presume  that  there  is  an  ability  to  relate  specific 
SUT-generated  state  estimates  to  specific  truth  states,  i.e.,  that  the  assoeiability  between  “Tracks”  (the 
SUT-generated  track  estimates)  and  “Truth”  (the  specified  Truth  states  for  the  given  T&E  experiments)  is 
known.  However,  conditional  on  many  factors  both  related  to  the  sensors  being  employed  as  well  as  the 
behavior  of  the  targets  and  also  the  specific  characteristics  of  the  various  algorithms  being  employed,  the 
ability  to  clearly  determine  which  SUT  Track  should  be  compared,  for  evaluation  purposes,  with  which 
Truth  track  may  often  be  unclear.  Such  ambiguities  in  DF-based  tracker  algorithm  evaluation  have  been 
known  and  flagged  as  evaluation  issues  as  far  back  as  the  90’s  (e.g.,  see  Refs  [1],  [2]).  However,  very 
few  papers  describing  techniques  to  deal  with  this  problem  when  evaluating  fusion-based  trackers  have 
been  published,  and  in  particular  almost  no  papers  (other  than  our  past  works)  have  been  published  that 
address  the  T&E  methodological  issues  related  to  this  problem. 

There  arc  a  number  of  types  of  tracker  algorithm  pathologies  that  can  arise  can  give  rise  to  the  Traek- 
to-Truth  association  problem.  A  few  cases  are  shown  below  in  Figure  2. 
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Figure  2.  Examples  of  pathologies  in  tracker  estimates. 


Most  of  these  problems  oeeur  as  a  result  of  mis-assoeiations  of  the  sensor  data  to  the  estimates  being 
generated  by  the  traeker  algorithm  (often  based  on  Kalman  Filter-based  techniques),  sueh  that  some  of  the 
sensor  data  for  given  targets  are  associated  to  another  target,  or  beeause  a  loeal  association  or  estimation 
error  creates  a  condition  where  the  track  is  lost,  or  another  example  is  when  two  (or  “n”)  targets  are  truly 
closely-spaced  and  sensor  resolution  limits  eouplcd  with  association  errors  result  in  traek  switching, 
where  the  estimation  process  mixes  estimates  for  multiple  targets  together  (this  can  occur  even  when 
target  identity  is  also  being  estimated  although  such  estimates  do  help  in  reducing  this  particular  type  of 
error).  A  wide  variety  of  other  difficulties  can  arise  even  with  the  most  sophisticated  and  modern  tracking 
techniques.  Thus,  tracker  evaluation  conditions  such  as  shown  in  Figure  3  can  arise: 
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“Tracks  (LI  Fusion)”  and  “Truth” 


Error  =  Estimated  -  Truth 


Truth  Track 


y 

Clearly  Depend  on  “Track-to-Truth”  Association 


Figure  3.  Notional  evaluation  case  for  fusion-based  target  tracking  process. 

The  point  of  this  diagram  is  that  in  order  to  assert  that  specific  tracking  errors  exist,  i.e.,  that  there  arc 
specific  differences  between  the  SUT  DF-based  track  estimates  (here  shown  in  black)  and  the  Truth 
tracks  (in  red),  an  assertion  of  which  SUT  track  goes  with  which  Truth  track  must  be  made.  In  the  face  of 
the  many  pathological  conditions  that  can  arise  (e.g.,  the  track  fragment  in  Figure  3,  as  well  as  the 
closeness  of  the  computed  tracks  to  the  Truth  tracks),  such  associations  arc  not  at  all  easy  to  assert.  As  the 
figure  indicates  on  the  right-hand  side,  a  many-to-many  association  problem  must  be  solved  to  assert  the 
SUT  Track  to  Truth  track  relationships  with  any  confidence.  The  specific  insertion  of  such  steps  is  one 
specific  recommendation  of  our  proposed  T&E  methodology.  Said  otherwise,  a  new  Data  Fusion  process 
must  be  designed  for  the  specific  purpose  of  testing  and  evaluating  any  DF-based  tracker.  (We  will  show 
later  that  this  is  a  requirement  for  any  DF  process,  including  the  higher  levels  of  fusion  (L2,  L3)  as 
previously  described.)  This  modified  approach  to  T&E  is  shown  notionally  in  Figure  4. 
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Figure  4.  Separate  Data  Fusion  processes;  the  SUT  DF  process  and  the  T&E  DF  process. 


It  should  first  be  mentioned  that  the  approach  to  constructing  any  Data  Association  (DA)  approach 
involves  three  sub-processes  as  shown  in  Figure  4:  Hypothesis  Generation,  where  the  feasible  causes  of 
any  given  observation,  to  include  Electronic  Warfare  techniques  i.e.,  deception,  are  defined.  Hypothesis 
Evaluation,  where  metrics  or  scores  that  reflect  the  degree  of  “closeness”  of  an  observation  to  an  estimate 
are  defined  and  calculated,  and  Hypothesis  Selection,  where  the  many-to-many  DA  or  “assignment”  type 
problem  is  solved,  culminating  in  an  optimal  assignment  of  the  observations  to  the  appropriate  state 
estimation  (fusion)  algorithms  for  each  target.  This  last  step  employs  what  are  called  “assignment 
algorithms”  imported  from  the  field  of  Operations  Research.  (The  term  “hypothesis”  here  means  and 
association  hypothesis,  i.e.,  a  nominated  observation-to-estimate  pairing.) 


Along  the  top  of  Figure  4  we  have  the  DA  and  DF  process  for  the  System  Under  Test  (SUT);  this 
process  notionally  uses  certain  Scoring  Metrics  and  Assignment  algorithms.  This  process  operates  on  the 
multi-sensor  input  stream  and  produces  target  track  estimates,  the  SUT  Tracks.  The  T&E  DA  process, 
notionally  employing  different  types  of  Scoring  Metrics  and  Assignment  algorithms,  takes  the  Truth 
tracks  as  the  definitive  associable  hypotheses,  and  calculates  the  “best”  assignments  of  SUT  Tracks  to 
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Truth  Tracks.  Given  those  assignments,  the  tracking  errors  (basically  grounded  on  differences  between 
the  estimated  states  and  the  Truth  states)  and  any  nominated  performance  metrics  can  be  estimated.  As 
pointed  out  in  Figure  4,  the  values  of  the  performance  metrics  clearly  depend  on  the  computed  Track-to- 
Truth  association. 

In  all  of  the  above,  we  have  been  emphasizing  that  the  DF  process  produces  estimates.  This  is 
because,  in  the  strictest  sense,  the  inputs  to  the  DF  processes  are  the  statistically-noisy  sensor  data  having 
stochastic  properties.  These  features  have  yet  other  implications  for  the  T&E  methodology,  namely  that 
the  stochastic  nature  of  the  process  needs  to  be  recognized  and  dealt  with  in  any  T&E  approach.  At  least 
when  conducting  any  simulation-based  T&E,  this  implies  that  (a)  the  experiments  should  be  designed 
through  the  employment  of  the  methods  of  statistical  experimental  design  (aka  Design  of  Experiments  or 
“DOE”),  and  (b)  in  conjunction  with  this  that  Monte-Carlo  based  replications  of  any  given  test  condition 
should  be  done. 

It  is  recognized  by  the  way  that  such  rigor  comes  at  a  price,  even  when  using  simulations,  and 
especially  w  hen  doing  field  tests  and  the  like.  It  is  likely  that  there  has  been  limited  application  of  these 
formal  methods  because  of  the  cost  implications.  As  academics  however,  we  feci  it  is  our  job  to  nominate 
rigorous  methods  so  that  their  application  can  at  least  be  assessed  in  any  given  case.  It  is  only  through  the 
use  of  such  methods  that  assertions  about  the  computed  metrics  can  be  made  with  statistical  confidence. 

There  are  other  issues  regarding  the  design  of  the  overall  T&E  process,  and  a  complete  discussion  of 
them  is  beyond  the  scope  of  this  paper.  To  give  one  example,  there  is  the  issue  of  the  design  of  the 
overall  T&E  process  for  a  typical  prototype  SUT  DF  process.  Any  given  real  DF  process  will  involve  a 
complex  processing  architecture,  not  a  single  DF  node,  because  the  design  of  such  real  systems  involves 
various  design  tradeoff  decisions.  A  typical  DF  process  architecture  may  appear  as  that  in  Figure  5. 


CUBRC  |  Developing  an  Approach  to  T&E  for  Data  Fusion  Processes 


m 

VISIBLE 

F 

m,f 

1 


IR 


mn 
r  M — m’f 


Imagery 


SAR  m  .  F  m,  F 

RADAR  F 

m-0-f 

i -  m  m,f,d  j 

EUNT  :■*—*  F 

♦  F 

f,d 

m, 

t - 1  m 

f 

f,d 

L.1 

f,d 

U 

Returns 

f,d 

KEY 

VIS  =  Visible  Wavelength  Sensors 

IR  =  Infrared  Sensors 

RADAR  =  Direct  Beem  Rader 
SAR  “  Synthetic  Aperture  Rader 
EL  =  Electronic  Intelligence  sources 
CO  *  Communications  Intelligence  sources 
HU  =  Human  Intelligence  sources 

F  =  Deta  Fusion  node 

m  =  Measurements  (pixels,  waveforms,  etc.) 
f  =  Features  (discrete  cr  continuous  ettributes) 
d  =  Decisions  (target  type,  location,  etc.) 


4  F 


COMINT 


f,d 


Tracks 

fd  LJ 

d 


f,d 


■  f 

f,d  -*f 

HUMINT 

F 

"1 

Text 

f,d 


L2 

d 


0 


LJ 

d 


Objects  Aggregates  Impacts 

•  Aircraft  •  Units  •  Objectives 

•  Ships  •  Subordi-  •  Vulnerabilities 

•  Vehicles  notion  •  Cost 

•  Cultural  &  *  Netting 
Natural  •  Logistics 
Features  •  Cross-Force 

•  -  etc. 


Figure  5.  Representative  Data  Fusion  processing  architecture. 


Here  wc  see  that  the  processing  flow  for  a  multiscnsor  system  involves  the  usual  batching  and 
partitioning  decisions  necessary  to  evolve  an  effective  and  efficient  processing  approach.  It  is  thus  typical 
that  tuples  of  sensor  groups  are  joined  in  local  DF  operations  (each  “F”  node  above  is  a  fusion  operation); 
this  is  done  for  various  reasons,  to  include  availability  of  the  data,  or  commonality  of  the  data,  etc.  The 
point  is  that  there  arc  in  any  system  multiple  DF  nodes  and  so  the  strategy  for  the  design  of  the  T&E  DF 
process  can  become  equally  complex.  Examples  of  the  various  strategics  that  can  be  applied  to  the  design 
of  the  T&E  DF  process  are  shown  in  Figure  6. 
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Source  Specific  Evaluation 


Time-Batched  Evaluation 


Figure  6.  Representative  design  choices  for  T&E  DF  nodal  processing. 

Here  we  see  that  the  T&E-specific  DF  processing  architecture  may  involve  choosing  DF  nodes  that  arc 
sensor  or  source  specific  (e.g.,  specific  to  a  radar  sensor),  or  a  strategy  that  is  time -based  where  Track-to- 
Truth  associations  and  estimates  of  metrics  are  computed  at  set  time  intervals,  or  a  strategy  that  is  event- 
driven  according  to  the  events  occurring  in  the  test  scenario.  Thus,  the  assertions  regarding  the  Track-to- 
Truth  relations  can  change  over  time,  or  change  according  to  the  flow  of  events,  or  according  to  yet  other 
choices  in  the  overall  T&E  DF  processing  architecture. 

2.2  Addressing  the  T&E  Process  for  Higher  Level  Data  Fusion-based  Tracking  Systems 

In  this  research,  we  are  now  looking  at  the  higher  levels  of  fusion,  involving  the  formation  of  Threat  or 
Risk  estimates  for  each  friendly  aircraft  in  these  scenarios,  as  developed  from  the  available  multi -sensor 
data.  In  this  case  we  are  exploiting  the  use  of  the  onboard  radars  and  the  Electronic  Support  Measures 
(ESM)  sensors  that  estimate  the  operating  modes  of  hostile  radars.  Conceptually,  the  Actual  Risk  to  a 
friendly  aircraft  can  be  thought  of  as  defined  by  the  relationship  between  an  Inherent  Risk  and  the  ability 
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to  thwart  that  risk  with  available  countermeasures.  The  way  this  notional  process  is  being  implemented  is 
shown  in  Figure  7. 


Figure  7.  Design  of  the  Actual  Risk  estimation  logic. 


For  the  eases  we  are  examining,  the  Inherent  Risk  can  be  estimated  by  estimating  the  Intent  and 
Lethality  of  a  given  hostile  platform.  In  turn,  with  the  available  sensor  suite,  we  can  employ  that  data  to 
estimate  Intent  by  examining  (a)  the  Mode  of  the  hostile  radar  (from  the  ESM  data)  and  (b)  the  relative  or 
Airborne  Intercept  aspect  or  inter-platform  geometry,  to  assess  for  example  whether  the  hostile  is  in  a 
shoot  geometry;  this  can  be  estimated  from  the  estimated  track  data  for  the  hostile  and  the  ownship 
navigation  data.  We  estimate  Lethality  using  a  concept  called  “Time  to  Lethal  Envelope”  or  TTLE,  also 
estimable  from  the  fused  kinematic  data,  and  representing  a  hypothetical  worst -ease  condition  where  both 
platforms  turn  directly  to  each  other,  with  the  TTLE  being  the  time  it  takes  to  get  within  maximum  hostile 
weapon  range. 

The  design  of  Airborne  Intercept  aspect  or  inter-platform  geometry  is  shown  in  Figure  9.  The  Airborne 
Intercept  is  calculated  based  on  the  truth  track  of  the  own  ship  (blue).  The  aspect  angle  is  the  angle 
formed  by  the  estimated  range  vector  with  the  estimated  velocity  vector  of  the  hostile  platform  (red).  The 
Airborne  Intercept  is  the  angle  formed  by  the  own  ship  velocity  vector  and  the  hostile  ship  velocity 
vector.  (Shown  as  Tue  Al  Aspect  Angle  C  and  Estimated  A1  Aspect  Angle  D  in  Figure  8) 
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Figure  8.  Air  Intercept  Aspect  Geometry 


To  calculate  the  Intent  the  possible  scenarios  should  be  considered.  These  scenarios  are  represented  in 


Figure  9.  Here  the  range  veetor  is  defined  to  origin  at  the  estimated  hostile  location  and  end  at  own  ship 


location.  The  range  veetor  angle  is  calculated  based  on  these  two  locations.  There  are  four  possible 
scenarios;  based  on  the  Al  Aspect  angle  formed:  less  than  90°;  between  90  and  180°;  between  180°  and 
270°  and  between  270*  and  360°.  There  are  two  possibilities  in  caeh  of  the  ease  and  the  range  angle  is 
used  to  differentiate  between  a  hostile  and  friendly  situation.  These  possibilities  are  shown  in  Figure  9. 
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Figure  9.  Air  Intercept  Aspect  Geometry  -  Possible  scenarios 


To  further  simulate  hostile  concept  of  employment  or  operation  (COE/COP)  for  hostile  radar,  we  use 


the  ESM  data.  The  ESM  data  is  estimated  using  the  estimated  velocity  vector  and  field  of  view  (FOV) 
[FOV  angle  is  a  user  input]  of  the  hostile  platform  (30°  around  the  velocity  vector).  In  Figure  10  we  see 
that,  hostile  H  can  see  platform  F|  while  F2  is  out  of  the  FOV  range  of  H. 


F, 


Figure  10.  Ownship/  Friendly  ESM 

There  are  four  possible  modes  simulated  for  the  hostile  radar: 

1 .  Unknown:  F  is  not  in  FOV  of  H 
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2.  Search:  When  F  is  in  H  FOV  and  both  platforms  arc  in  an  approaching  or  closing  kinematic 
relationship. 

3.  Track:  When  F  is  in  search  mode  for  At!  time. 

4.  Lock-on:  (Figure  11)  When  F  is  in  Track  on  mode  for  At2  time  and  also  reaches  within  the 
Hostile  missile  range  (R^*) 


Figure  11.  Lock-on  Mode  Declaration  Logic 

The  ESM  sensor  is  assumed  to  have  long  range,  4n  sensitivity.  Being  passive,  the  correct  calculation 
for  the  condition  that  the  Friendly  has  ESM  data  available  to  it  is  based  on  whether  the  Friendly  is  truly  in 
the  FOV  of  the  Hostile  radar  (we  assume  the  H  ConOp  employs  the  radar  actively).  Thus,  the  geometric 
calculations  for  FOV  containment  are  based  on  the  Hostile  Truth  track  data.  To  generate  True  Mode 
according  to  this  logic,  go  to  Confusion  matrix  to  generate  Actual  Mode  Declaration  Report  (Tabic  1). 


Table  1  .  Actual  Mode  Declaration  Report. 


Actual  Mode  Declaration 

True  Mode  (Geometry,  Time) 

Search 

Track 

Lock -On 

Search 

P(S|S) 

P(S|T) 

P(S|LO) 

Track 

P(T|S) 

P(T|T) 

P(T|LO) 
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Lock-On 

P(LOiS) 

P(LO|T) 

P(LO|LO) 

Unknown 

P(U|S) 

P(U|T) 

P(U|LO) 

To  simulate  the  lethality,  we  use  the  concept  of  “Time  to  Lethal  Envelope”  (TTLE).  TTLE  is  the 
hypothetical,  worst-case  calculation  which  assumes  that  both  hostile  and  friendly  platforms  turn  directly 
toward  each  other  at  max  velocity.  TTLE  is  the  time  to  close  to  within  Hostile  missile  launch  maximum 
range  depicted  in  Figure  12. 


TTA-based 
Hostile  Truth  Track 


SUT-estimated  Hostile  Track 


Figure  12.  Time  to  Lethal  Envelope  (TTLE) 

These  Intent  and  Lethality  estimates  can  be  logically  fused  to  assert  a  level  of  Inherent  Risk.  The  top- 
level  approach  is  shown  in  Figure  13.  Here  wc  see  that  the  friendly  platform  sensor  data  are  fed  to  the 
Level  1  SUT  DF  process  which  provides  the  fused  kinematic  state  estimates  on  all  hostile  platforms.  The 
ESM  data  are  used  to  estimate  the  RF  Mode  of  the  hostile  radars  and  the  fused  kinematic  estimates  are 
used  to  construct  the  Level  2  SUT  DF  estimate  of  the  “situation”,  and  then  the  situational  estimates  are 
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fused  using  the  logie  of  Fig.9  to  provide  the  Level  3  fused  estimate  of  Inherent  Risk.  We  are  using  a 
Fuzzy  Logie  approach  to  forming  the  L3  DF  logie  for  the  reasons  shown  in  Figure  13. 


L2,  L3  SUT  Process  Design 


Sensors 


Estimated  Hi  RF  Mode 
from  Friendly  J 

Estimated  Hi  Ai  Aspect 
from  Friendly  J 

Estimated  Hi  TTLE 
from  Friendly  J 


Attributes 


L2  -  (Relations  between  SUT  Hi,  FJ> 
“L2  Tracks” 


L3  -  (inherent  Risk  (Threat)  relations  between  SUT  HI,  FJ> 
“L3  Tracks” 


Rationale 

—can  deal  best  w  vague  data 
—quantitative 

—existing  software,  experience 
—computationally  fast 


Figure  13.  Top-level  approach  to  higher-level  fusion  process  design. 

There  are  various  design  options  available  to  fuse  the  L2  SUT  estimates  to  get  L3  SUT  estimate  of 
Inherent  Risk;  like  Bayesian  Networks,  Figure  of  Merit,  ete.  Here  we  use  Fuzzy  Logie  to  fuse  the  L2 
estimates.  Fuzzy  Logie  (FL)  is  an  infereneing  methodology  that  is  directed  toward  vague  relationships 
between  evidence  and  assertions.  Using  natural  language  statements  that  eontain  appropriately-vague 
terms  (e.g.,  “elose”)»  FL  provides  a  quantitative  framework  for  relating  the  interdependent  phrases  of 
these  expressions.  Fuzzy  inference  is  the  proeess  of  formulating  the  mapping  from  a  given  input  to  an 
output  using  FL.  Fuzzy  inference  systems  that  have  been  successfully  applied  in  fields  sueh  as  automatic 
eontrol,  data  classification,  decision  analysis,  expert  systems,  and  computer  vision.  Beeause  of  its 
multidisciplinary  nature,  fuzzy  inference  systems  are  associated  with  a  number  of  names,  sueh  as  fuzzy- 
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rule-based  systems,  fuzzy  expert  systems,  fuzzy  modeling,  fuzzy  associative  memory,  fuzzy  logie 
controllers,  and  simply  (and  ambiguously)  fuzzy  systems. 

Fuzzy  Logie  is  a  useful  tool  in  making  decisions  in  light  of  information  that  is  impreeise  and 
incomplete.  Given  information  eolleeted  from  multiple  sourees  sueh  as  a  target's  location,  aspeet,  and 
speed,  FL  ean  be  used  to  measure  the  degree  of  danger  of  the  target  without  formulating  eomplex 
mathematical  equation.  The  FL  functions  are  more  natural  for  the  representation  of  the  feeling  of 
ineertitude.  A  very  preeise  information  is  not  expeeted  absolutely,  but  hope  for  the  greatest  possible 
eoherenee.  On  the  other  hand,  preeise  but  fluctuating  data  more  usually  result  from  the  observation  of  a 
physical  phenomenon. 

Given  information  eolleeted  from  multiple  sourees  sueh  as  a  target’s  ESM  Mode,  Air  Intercept,  and 
TTLE,  we  measure  the  degree  of  Inherent  Risk  of  the  target  by  adopting  FL  easily  without  formulating 
the  complex  mathematical  equation.  In  this  research,  we  adopt  FL  for  measuring  the  Inherent  Risk 
beeause  of  a  eouple  of  reasons.  In  certain  observation  and  reporting  eireumstanees,  it  may  not  be 
appropriate  to  represent  those  variables  in  the  probability  domain.  That  is,  the  state  of  these  variables  does 
not  have  an  associated  “erisp”  set.  For  eonvenienee,  we  suppose  that  the  three  SUT  estimates  have  three 
states  respectively;  these  are  the  type  of  representations  that  would  eome  from  a  reporting  or  message- 
based  input,  rather  than  from  the  sensors  themselves.  Unlike  the  variables  in  Bayesian  Networks,  whieh 
have  erisp  set,  these  Aspeet  and  TTLE  variables  have  fuzzy  sets. 

Consider  an  example  of  measuring  the  height  of  all  the  children  in  a  elass.  How  ean  we  classify  the 
height  of  the  children  as  low,  medium  or  high?  If  we  say  a  person  having  height  greater  than  6  feet  is  tall, 
then  we  have  a  situation  where  a  5.9  feet  person  terms  out  to  be  of  medium  height  and  a  person  with 
height  of  6.1  feet  turns  out  to  be  in  tall  group.  But  in  FL  a  membership  function  is  used  to  measure  the 
degree  of  membership  of  the  quantitative  value  (Here,  height  of  a  ehild)  in  a  fuzzy  set. 

Fuzzy  Logie  also  has  advantage  in  representing  of  kinematie  and  angle  data  sinee  it  uses  natural 
language.  The  fuzzy  inference  system  has  membership  functions,  fuzzy  logie  operators  and  if-then  rules. 
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There  are  two  types  of  fuzzy  inference  systems:  Mamdam-type  and  Sugeno-type.  The  Mamdani  type 
inference  system  has  been  adopted  in  the  framework  for  L3  fusion  that  is  reported  herein. 


Table  2.  Fuzzy  Logic  -  Threat/Risk  Logic  Rules 
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13 

med 

large 

Track 

med 

27 

high 

low 

Lock 

high 

14 

med 

med 

Track 

high 

Note  that  the  L3  SUT  DF  process  can  be  thought  of  as  producing  “Inherent  Risk  Tracks”,  i.e.  timewise 
histories  of  the  estimated  level  of  Inherent  Risk,  conceptually  as  much  as  track  as  the  timewise  position 
histories  of  a  physical  platform.  Note  too  that  the  issue  of  associating  estimated  Inherent  Risk  tracks-to- 
Inherent  Risk  Truth  tracks  will  be  a  challenge  in  designing  the  new  T&E  DF  process  to  evaluate  these 
new  estimates.  The  Inherent  Risk  Truth  tracks  are  those  computed  by  using  the  truth  values  of  kinematics 
etc  in  the  track  formation. 

2.3  Addressing  the  T&E  Process  for  Level  4  Data  Fusion-based  Tracking  Systems 

That  risk  would  be  mitigated  according  to  the  possible  employment  of  Countermeasures  (CM) 
available  to  the  friendly  platform.  Continuing  from  the  T&E  framework  for  Level  2  and  3  Fusion  process 
we  employ  the  Level  4  Fusion  process  as  shown  in  Figure  14.  Here  wc  employ  electronic  countermeasure 
(ECM).  Any  electronic  effort  which  intends  to  disturb  normal  radar  operation  is  referred  to  as  ECM.  ECM 
are  employed  to  accomplish  improper  or  delayed  target  detection,  analyst  deception  or  generate  false 
positives.  There  are  two  classes  to  CMs,  one  which  actively  deny  radars  to  perceive  an  measurement  like 
Jammers,  and  deceptive  CMs  like  changing  trajectory. 
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Figure  14.  Top-level  approach  to  level  4  fusion  process  design 


Here  we  simulate  the  situation  where  the  own  ship  platforms  have  Self-Sereening  Jammers  (SSJ). 
They  are  type  of  self  protecting  jammers  whieh  earry  a  jammer  on  the  platform  like  beam  jammers. 
Another  type  of  jammer  not  implemented  here  is  Stand-off  jammer  whieh  requires  an  eseort  vehiele 
whieh  earries  the  jammer.  The  details  about  the  jammers  parameters  were  referenced  from  Mahafza  and 
Elsherbeni  [49].  Here  are  some  of  the  terms  relevant  to  the  CM  implementation  [49]: 

Pt  =  Peak  Power  in  Watts  =  =  50.0eK3 

G  =  Antenna  Gain  in  dB  =  35.0 
X  =  Antenna  wavelength 
Ar  =  Antenna  aperture 
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o  =  10.0;  %  Radar  Cross  Seetion  in  m  sqaured 


Br  =  Radar  Operating  Bandwidth  in  Hz  =  667.0e+3 

L  =  Radar  Losses  in  dB  =  0. 1000 

Pj  =  Jammer  Peak  Power  in  Watts  =  200.0 

Bj  =  Jammer  Operating  Bandnwidth  in  Hz  =  50.0e+6 

Gj  =  Jammer  Antenna  Gain  in  dB  =  10.0 

Lj  =  Jammer  Losses  in  dB  =  0. 1 0 

R  =  Range 

A  single  pulse  power  received  by  the  radar  from  target  is  given  as: 

PtG2X2pr 
5  "  (4 tt)3/?4L 


The  power  received  by  the  radar  from  a  SSJ  jammer  at  same  range  is  given  as: 


i  = 


PjGjAr 

4nR2BjLj 


Where  Ar  =  —  then  ]  = 


P/C//t2C 
(4; tR)2BjL}' 


A  jammer  ean  be  identified  by  its  effective  operating  bandwidth 


and  its  Effective  Radiated  Power  (ERP ):ERP 


.  Then  the  Signal  to  Jammer  ratio(S/J)  is  given  as: 


5 

7 


Pt  *  (low)  *  a  *  Bj  *  (10^) 


4.0  *  7T  *  Pj  *  ( 10To  J  *  Br  *  ( lOTEf)  *  ((/?  *  1000.0)2) 


O') 
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S/J  tells  what  the  effect  of  the  jammer  on  the  radar  is.  The  jamming  power  from  the  platform  is  a  one¬ 
way  transmission  while  the  target  signal  power  is  a  two  way  transmission.  So,  the  jamming  power  is 
generally  greater  than  target  signal  power  (S/J  <  1).  But  as  the  target  comes  closer  there  will  certain  range 
at  which  S/J  ratio  equals  1 ,  and  it  is  called  as  cross-over  range  and  is  given  as: 


Cross  Over  Range 


Pt*  1 

(low) 

\  *  a  *  Bj  *\ 

1  \ 

'io&) 

|  *Br 

■  *  1 

(lo4)J 

1000.0 


So,  to  remain  undetected  the  platform  has  to  remain  at  a  range  higher  than  cross-over  range.  The  range 
at  which  the  radar  can  detect  and  perform  proper  measurements  for  a  given  S/J  is  called  as  bum -through 
range.  So  the  logic  on  CM  implemented  here  is  that  the  own  ships  try  to  maintain  safe  distance  from  the 
hostile  targets.  The  platform  jamming  power  is  in  proportion  to  the  target  signal  power  until  the  platform 
reaches  the  cross-over  range.  When  the  platform  reaches  the  cross-over  range  and  is  unable  to  block  the 
target  signal,  the  second  counter  measure  logic  sets  in.  The  ownship  platform  will  change  its  flight  plan 
by  changing  the  flying  trajectory  by  a  preset  angle  and  try  to  move  out  of  the  hostile  field  of  view  (FOV). 


3  Case  Study:  PE  Simulator  for  AFFTC 

In  our  earlier  work,  we  have  summarized  some  of  the  generalized  issues  when  considering  the  test  and 
evaluation  of  a  prototype  data  fusion  process  (what  we  have  called  the  “System  Under  Test”  or  SUT). 
We  focused  on  the  problem  of  PE  and  the  “fairness”  issue  for  the  distributed  data  fusion  case.  In  this 
AFFTC  application,  one  type  of  distributed  fusion  application  will  be  the  case  of  multiple  aircraft 
platforms  working  cooperatively  on  a  common  mission,  each  performing  local  or  platform-specific  fusion 
while  also  exchanging  data  and  fused  estimates  to  each  other.  A  core  evaluation  issue  herein  is  the 
assessment  of  the  degree  of  consistency  in  the  multiple  track  pictures  across  the  platforms.  It  is  also 
critical  to  guarantee  that  the  alternative  PE  network  node  outputs  are  consistent,  in  accordance  with  a 
consistency  specification  pertinent  to  the  application.  PE  nodes  perform  track-to-truth  association  to 
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support  track  accuracy-related  or  other  MoPs,  and  perform  track-to-track  association  to  support  platform 
track  file  consistency-related  MoP  estimation  for  two  or  more  intemetted  platforms  (e.g.,  Joint  Strike 
Fighters  (JSFs),  or  the  “F-35”  aircraft). 

3.1  PE  Node  Design 

In  the  PE  framework  the  PE  nodes  perform  3  necessary  functions:  (i)  data  preparation  (ii)  data 
association  and  (iii)  MoP  state  estimation.  In  our  Case  Study,  during  data  preparation  the  PE  node  puts 
tracks  and  truth  information  in  [jc,  y]  co-ordinates  and  common  time.  Data  association  performs 
deterministic  track-to-truth  association  and  track-to-track  association.  During  data  association  the 
following  three  actions  are  performed: 

(i)  Hypothesis  Generation, 

(ii)  Hypothesis  Evaluation,  and 

(iii)  Hypothesis  Selection. 

The  PE  node  uses  a  Kalman  filter  for  Level  1  MoP  state  estimation.  Using  the  Level  1  estimates 
the  Level  2  and  3  estimates  are  generated.  SUT  tracking  errors  will  induce  Intent  (AI  Aspect  and  ESM 
mode)  and  Lethality  (TTLE)  errors.Thcse  errors  will  propagate  to  a  component  error  in  Threat/Risk 
assessment.  Note  that  these  arc  different  than  Threat  errors  derived  from  Threat -to-T ruth  Threat 
Association. 

3.2  Case  Study  Measures  of  Performance  for  PE 

This  overall  T&E  methodology  has  been  applied  to  some  cases  of  interest  to  the  Air  Force  that  involve 

DF-capable  tactical  aircraft.  (Many  modern-day  tactical  aircraft  have  multiple  sensor  systems  and  can 
employ  DF  processes  to  support  the  execution  of  their  missions.)  Because  certain  aircraft  employment 
concepts  involve  multiple  aircraft  cooperating  on  a  given  mission,  wc  have  recently  examined  a  case 
involving  two  friendly  aircraft  engaged  in  an  air-to-air  combat  environment  with  six  hostile  aircraft.  From 
a  Data  Fusion  point  of  view,  this  involved  addressing  the  T&E  issues  associated  with  Distributed  Data 
Fusion  (DDF),  since  fusion  occurs  not  only  on  each  friendly  aircraft  but  also  between  them,  since  they  are 
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intercommunicating,  and  exchanging  both  data  and  fused  estimates  in  a  rather  complex  DDF  process. 
The  framework  of  these  experiments  is  shown  in  Figure  15. 


Figure  15.  Framework  of  Distributed  Data  Fusion  experiments. 


From  the  point  of  view  of  supporting  the  tactical  mission,  one  critical  issue  of  course  if  whether  there 
is  a  consistent  “track  picture”  across  the  two  aircraft.  It  can  be  seen  in  Figure  15  that  it  is  typical  that 
there  are  differences  in  the  local  target  track  pictures  on  each  platform  which  need  to  be  reconciled  for 
mission  application.  In  these  recent  studies  then,  wc  studied  fused  track  picture  consistency  as  a  function 
of  certain  factors,  looking  at  both  Track-to-Truth  and  Track-to-Track  consistency  metrics. 

Figure  15  depicts  how  the  two  platforms  have  their  own  view  of  the  truth  picture  based  on  the  on¬ 
board  sensors.  There  are  both  “common”  pictures  and  “unique”  pictures.  Let  us  assume,  for  the  sake  of 
example,  that  all  the  on-board  sensors  see  the  same  targets.  Let  platform  1  sees  3  tracks  (based  on  on¬ 
board  sensors)  which  are  common  to  platform  2  and  vice  versa.  The  common  tracks  are  shown  in  red. 
Note  that  even  though  both  of  the  platforms  sec  the  same  targets,  their  measurements  about  those 
common  targets  could  be  different  depending  on  how  the  on-board  sensors  report  the  measurements.  Also 
there  are  certain  targets  that  are  uniquely  seen  by  platform  1  and  platform  2;  note  that  some  of  either  the 
common  or  unique  tracks  could  be  false  tracks. 

Lach  of  the  platforms  exchange  their  track  files  and  data  fusion  is  done  upon  receipt  of  this 
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information  at  each  platform.  We  will  explain  further  how  this  information  is  exchanged  when  we  diseuss 
Tier  0,  Tier  I  and  Tier  2  (Section  3.3).  We  assume  that  there  is  no  bandwidth  limitation  in 
communication.  We  have  incorporated  more  realistic  asynchronous  (delayed)  communication  among  the 
sensors  and  the  platforms. 

The  baseline  distributed  fusion  output  is  the  Consistent  Taetieal  Picture  (CTP).  The  sensor  track  file 
“consistency”  is  computed  at  each  time  point  as  the  percentage  of  matching  CTP  traeks  in  the  track  files 
of  eaeh  platform.  In  addition  to  this  measure,  the  following  four  higher  level  consistency  metrics  have 
been  computed; 

7.  Track-to-Track  Aspect ,  ESM  and  TTLE  Consistency :  These  are  Level  11  metrics  which  compares  the 
Air  Intercept,  ESM  mode  and  TTLE  estimates  across  platforms. 

2.  Track-to-Truth  Aspect,  ESM  and  TTLE  Consistency :  These  are  Level  11  metrics  which  measure  the 
accuracy  of  the  Air  Intercept,  ESM  mode  and  TTLE  estimates. 

2.  Track-to-Track  Threat  Consistency :  This  is  a  Level  III  metric  which  compares  the  estiamted  Threat 
across  platforms. 

4.  Track- to- Truth  Threat  Consistency :  This  is  a  Level  111  metrie  whieh  measures  the  accuracy  of  the 
estimated  Threat. 

3.3  Explanation  of  Fusion  Tiers 

Tier  0:  (Figure  16)  In  Tier  0,  each  of  the  on-board  sensors  (Radar,  ESM  and  1RST)  fuse  their  own 
reports.  The  resultant  Tier  0  tracks  are  then  fused  together  to  get  the  Tier  I  consistent  track  picture.  Here 
the  information  is  not  yet  shared  aeross  the  platforms,  so  the  result  tends  to  be  less  aeeurate  than  for 
example  the  fusion  of  Tier  0  sensor  traeks  to  the  all  source  CYP.  Generally,  batehing  of  larger  data  sets 
for  fusion  is  more  accurate;  albeit  more  complex. 
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Figure  16.  Tier  0 


Tier  1:  (Figure  17)  In  Tier  1,  each  of  the  on-hoard  sensors  (Radar,  ESM  and  1RST)  share  their  Tier  0 
track  files  to  generate  the  ownship  consistent  track  picture.  This  is  typically  done  for  eaeh  sensor  traek  file 
as  it  is  updated,  rather  than  all  sensors  at  onee.  The  DNN  architecture  exposes  these  and  many  other  ways 
to  network  fusion  nodes  on  a  single  platform  for  Tier  1  fusion  or  on  multiple  platforms  for  Tier  2  fusion. 


Figure  17.  Tier  1 

Tier  2:  (Figure  18)  In  a  typical  Tier  2  fusion  the  Tier  1  traek  files  are  fused  sequentially  as  eaeh  Tier  1 
traek  file  is  updated.  A  modified  form  of  a  Tier  2  fusion  network  is  for  eaeh  platform  to  share  its  own 
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sensor  measurements  with  the  other  platforms.  This  can  be  done  one  sensor  at  a  time  sequentially  as  each 
sensor  scan  of  data  is  received.  This  alternative  tends  to  be  more  accurate,  however  at  a  cost  of  more 


communications  bandwidth  and  fusion  complexity  (e.g.,  due  to  report  propagations  for  time  delays, 
multiple  platform  coordinate  misalignments,  intemetted  ghost  tracks,  etc.). 


Figure  18.  Tier  2 

4  Experimental  Results  and  Analysis 

The  baseline  2  vs.  6  offensive  sweep  scenario  has  6  foe  fighters  (targets)  engaging  simultaneously  in 
pairs  from  left  and  right  45  degrees  and  center  to  achieve  a  simultaneous  missile  launch  against  the  blues 
(platforms).  The  blue  1  launches  AMRAAM  missiles  on  reds  (fighters)  1,  2  and  3,  4  respectively.  The 
blue  1  launches  AMRAAM  for  the  second  time  against  the  surviving  red.  Then  the  other  blue  turns 
towards  reds  5,  6  and  launches  missile.  All  the  red  fighters  are  in  a  pair  staggered  formation  with  the 
trailing  fighter  off  to  the  left  or  to  the  right,  sufficient  to  be  not  resolvable  by  blue  radar  until  after  the 
final  red  turn. 


Tier  0:  We  ran  the  simulation  for  all  Tiers  from  time  periods  1  to  329  with  an  interval  of  1  time  period. 
The  time  period  was  1  second.  The  baseline  2vs6  offensive  sweep  scenario  has  6  foe  fighters  coming 
towards  2  blue  fighters  with  the  objective  of  engaging  at  10-15  km  simultaneously  in  pairs  from  ±45 
degrees  and  center.  The  blue  launch  AMRAAM  missiles  between  20-25  km  on  1,  2  and  3,  4,  respectively. 
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The  second  launch  by  blue  1  against  the  surviving  red  3  oeeurs  at  about  10-15  km.  Then  the  other  blue 
turns  at  5g  towards  red  5,  6  and  launches  on  5,  6.  All  fighters  are  in  a  pair  staggered  formation  with  the 
trailing  fighter  off  to  right  and  behind  sufficient  to  be  not  resolvable  by  blue  radar  until  after  the  final  red 
turn. 

The  blue  and  red  fighters  are  both  initially  in  search  mode  for  each  other.  Once  the  reds  detect  they 
turn  off  emissions  and  execute  their  pre-planned  maneuvers  to  achieve  near  simultaneous  launch  on  the 
projected  blues.  The  reds  all  turn  on  their  radars  to  lock-on  to  blues  just  after  their  last  turn  towards  the 
projected  blue  position.  The  reds  launch  radar  guided  missiles  at  their  closest  blue  targets  as  soon  as 
possible.  Red  5/6  should  pull  delaying  turns  together  then  turn  towards  an  intercept  with  US  1  (i.c., 
highest  closure  rate)  once  their  radar  acquires. 

The  blues  split  and  turn  towards  the  outside  threats  to  take  advantage  of  their  longer  range 
AMRAAM  shots  at  each  of  outside  red  pairs.  They  support  their  launches  until  both  outside  reds  are 
killed  or  until  second  shots  are  needed.  In  the  baseline  scenario  shown,  US2  achieves  2  kills  with  its  first 
launches  then  turns  towards  reds  5/6  that  have  engaged  US1  while  taking  its  second  shot  at  the  surviving 
red  4.  US1  will  leave  this  second  AMRAAM  onee  it  has  acquired  red  4,  then  pulls  defensive  maneuvers 
and  countermeasures  against  the  reds  5/6  missile  launches  while  US2  completes  red  5/6  kills. 

The  SUT  gate  multiplication  factor  was  5  and  15.  The  PE  gate  multiplication  factors  of  3  and  5,  PE 
designs  for  Vogel  and  Hungarian  based  association,  expected  probability  of  false  tracks,  expected 
probability  of  detection  and  confidence  ID  updates. 

Tier  1:  Similar  to  Tier  0,  the  simulation  for  Tier  1  was  run  from  time  periods  1  to  329  with  an  interval  of 
1  time  period. 

Tier  2:  The  simulation  for  Tier  2  was  run  from  time  periods  1  through  329  at  an  interval  of  1  time  period. 

5  Design  of  Experiments 

When  employing  DOE  test-planning  methods,  one  issue  that  can  arise  is  the  complexity  involved  in 
designing  efficient  test  plans  if  there  are  many  independent  variables  (or  “factors”,  the  term  used  in  the 
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DOE  literature)  whose  effects  on  the  DF  process  under  test  want  to  be  known.  Using  traditional  DOE 
experimental  designs,  the  number  of  runs  that  have  to  be  made  will  grow  exponentially  when  the  number 
of  factors  is  large,  and  the  number  of  “levels”  (specific  value  settings  of  the  factors)  is  large  ;  these  go  as 
the  number  of  levels  raised  to  the  number  of  factors,  or  LF.  This  exponential  growth  is  associated  with  the 
type  of  experimental  design  being  employed,  called  a  “factorial”  design,  which  not  only  allows  the  so- 
called  “main  effects”  to  be  discerned  from  the  experiments  but  also  what  are  called  “interaction”  effects, 
where  knowledge  is  gained  about  the  effects  on  the  metrics  of  interest  due  to  interacting  effects  among 
the  factors.  If  the  desire  to  learn  about  the  interaction  effects  is  relaxed,  using  a  type  of  experimental 
design  called  a  “fractional  factorial”  design,  the  severity  of  the  exponential  growth  is  lessened  but  can  still 
be  an  issue  to  deal  with  Although  we  are  still  studying  the  strategies  for  and  efficiencies  of  large  factor- 
many  level  experimental  designs,  we  are  now  employing  a  phased  approach  as  shown  in  Figure  19,  where 
we  use  the  fractional  designs  initially  as  a  screening  step  to  determine  those  factors  which  are  most 
influential  on  the  metrics,  and  then  the  factorial  designs  to  better  understand  the  main  and  interaction 
effects  of  the  key  variables  and,  if  necessary  what  are  called  “response  surface”  methods  to  understand 
the  broad  effects  of  the  factors  across  the  levels  of  interest  for  the  application. 
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One  DOE  Strategy  for  Many- factor  Cases 


Focus  of  Analysis 


Figure  19.  Phased  experimental  design  strategy  to  deal  with  many  factors  and  levels. 


Perhaps  the  most  important  aspect  of  this  formalized  approach  is  that  the  post-test  analysis 
procedures,  generally  falling  under  the  title  of  “Analysis  of  Variance”  or  “ANOVA”  procedures,  allows 
the  assessments  of  the  results  to  be  done  with  statistical  significance.  That  is,  assertions  of  the  type  that 
“the  hypothesis  that  there  is  an  effect  of  factor  X  on  metric  Y  can  be  rejected  with  95%  statistical 
confidence”  can  be  made  as  a  result  of  the  combined  utilization  of  DOE  test  designs  and  ANOVA 
analysis  methods.  We  planned  a  Design  of  Experiments  (DOE)  scheme  for  the  PE  MoPs.  We  conducted 
these  tests  on  Tier  0,  Tier  1  and  Tier  2.  We  decided  on  the  following  factors  to  setup  the  DOE: 

•  Scenario  Factors  (Fixed): 

-  Offensive  Sweep  2vs6  Air-to-Air 

•  PE  Factors: 

-  Design  (Association) 

Vogel  Approximation  (PE  1),  and 
Hungarian  based  association  (PE  2) 
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Gating  Factor:  3  and  5 


•  System  under  test  (SUT)  Design  Factors: 

-  Gating  Factor:  5  and  15 

So  this  yields  a  2k  or  23  full  factorial  design.  We  used  M1N1TAB  to  perform  the  DOE  runs.  The  full 
factorial  design  details  are  as  follows: 

Factors:  3 
Levels:  2 

(A)  SUT  Design  Gating  Factor 

(B)  PE  Gating  Factor 

(C)  PE  Design 

Base  Design:  3,  8 
Runs:  80 
Replicates:  10 
Blocks:  1 

Center  pts  (total):  0 

All  terms  are  free  from  aliasing.  The  factors  and  interactions  that  are  significant  for  various  MoPs  are 
denoted  by  ‘S’.  Table  3,  Table  4  and  Tabic  5  show  the  summary  of  the  DOE  run  results  for  Tier  0,  1  and  2 
respectively.  In  addition  to  these  DOE  runs,  we  ran  another  set  of  full  factorial  runs  to  see  the  effect  of 
communication  tiers  on  the  various  MoPs.  We  added  another  factor,  (D)  Tier,  with  two  levels:  Tier  1  and 
Tier  2.  Tabic  6  shows  the  significant  factors  and  their  interactions  for  the  various  MoPs.  The  detail  DOE 
charts  are  given  in  Appendix  A,  B,  C  and  D. 


Table  3.  Tier  0  DOE  run  summary 


A 

B 

C 

AB 

AC 

BC 

ABC 

Track  1  to  Truth  Radar 

Aspect  Consistency 

ESM  Consistency 
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TTLE  Consistency 

Threat  Consistency 

s 

Track  2  to  Truth  Radar 

Aspect  Consistency 

s 

ESM  Consistency 

s 

s 

TTLE  Consistency 

s 

s 

s 

Threat  Consistency 

s 

Track  1  to  Truth  ESM 

Aspect  Consistency 

s 

s 

ESM  Consistency 

TTLE  Consistency 

s 

s 

Threat  Consistency 

s 

s 

s 

Track  2  to  Truth  ESM 

Aspect  Consistency 

ESM  Consistency 

s 

s 

TTLE  Consistency 

s 

s 

Threat  Consistency 

s 

Track  1  to  Truth  IRST 

Aspect  Consistency 

ESM  Consistency 

s 

TTLE  Consistency 

s 

Threat  Consistency 

Track  2  to  Truth  IRST 

Aspect  Consistency 

ESM  Consistency 

TTLE  Consistency 

Threat  Consistency 

s 

Table  4  .  Tier  1  DOE  run  summary 


A 

B 

c 

AB 

AC 

BC 

ABC 
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Track  to  track 

Aspect 

ESM 

TTLE 

Threat 

s 

Track  1  to  truth 

Aspect 

s 

s 

ESM 

s 

TTLE 

Threat 

Track  2  to  truth 

Aspect 

s 

s 

ESM 

s 

TTLE. 

s 

s 

Threat 

s 

Table  5  .  Tier  2  DOE  run  summary 


A 

B 

c 

AB 

AC 

BC 

ABC 

Track  to  track 

Aspect 

S 

ESM 

TTLE 

S 

Threat 

S 

Track  1  to  truth 

Aspect 

S 

S 

ESM 

s 

s 

S 

S 

S 

TTLE 

s 

s 

S 

Threat 

S 

Track  2  to  truth 

Aspect 

s 

ESM 

s 
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TTLE 

Threat 

s 

s 

Table  6  .  Inter  Tier  1  and  2  DOE  run  summary. 


A 

B 

c 

D 

AB 

AC 

AD 

BC 

BD 

CD 

ABC 

ABD 

ACD 

BCD 

ABCD 

Track 

to  track 

Aspect 

ESM 

S 

s 

s 

S 

s 

TTLE 

s 

Threat 

s 

S 

Track  1 

to  truth 

Aspect 

s 

s 

s 

s 

s 

s 

S 

s 

S 

s 

ESM 

s 

s 

s 
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The  concept  of  analyzing  the  design  for  L4  design  is  really  difficult.  According  to  the  Countermeasure 
logic  implemented  in  Section  2.3,  the  estimated  tracks  generated  will  be  significantly  different  than  the 
truth  track.  The  countermeasure  logic  suggests  that  jam  the  radar  signal  from  the  hostile  platform,  until 
the  hostile  reaches  the  cross-over  range  (S/J  =1).  After  that  the  ownship  changes  its  current  trajectory  to 
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get  out  of  the  FOV  of  the  hostile  platform.  Due  to  this  countermeasure  implementation  the  true  threat  will 
be  different  that  the  estimated  threat  and  will  hence  forth  have  different  tracks.  To  check  for  the  change  in 
S/J  consistency  over  the  truth  and  estimated  tracks  and  the  DOE  charts  arc  shown  in  Appendix  E. 

The  statistical  design  shown  here  deals  with  very  small  number  of  factors.  But  in  case  of  large  number 
of  factors  we  will  need  humongous  number  of  runs  to  analyze  the  statistical  design  under  study.  We  need 
to  change  our  approach  or  find  a  new  method  to  handle  the  increasing  number  of  factors.  In  following 
section  we  have  elaborated  large  factor  DOE  (Design  of  Experiment). 

6  Large  Factor  DOE 

Here  we  are  trying  to  achieve  a  tradeoff  between  the  number  of  factors,  their  levels  and  the  cost  of 
carrying  out  the  experiment.  As  the  former  increase,  the  cost  increases.  In  a  full  factorial  design  we 
analyze  all  the  factors  at  all  levels  which  have  large  number  of  runs.  For  example  in  a  2  level  full  factorial 
design  with  5  factors  a  minimum  of  32  runs  need  to  be  carried  out.  This  makes  it  impractical  to  carry'  on 
with  such  an  experiment.  Generally  this  is  the  major  reason  experimenters  opt  for  a  fractional  factorial 
design  where  information  is  obtained  very  easily  and  in  fewer  runs.  But  it  is  a  highly  confounded  design. 
A  lot  of  information  is  lost  because  of  this  confounding.  As  interactions  are  confounded  with  the  main 
effect  it  becomes  very  unclear  as  to  which  is  the  factor  that  is  responsible  for  the  effect. 

Goals  of  large  factor  design  In  order  to  overcome  the  shortcomings  of  a  fractional  factorial  design, 
certain  designs  tend  to  the  large  number  of  factors  and  are  economical  and  produce  optimal  results  in  very 
less  time.  The  goals  of  such  designs  can  be  listed  as: 

•  Lower  number  of  trails  with  variations  of  multiple  factors. 

•  Finding  the  separation  of  effects  due  to  individual  factors  and  interactions. 

•  Keeping  a  vertical  balance 

These  designs  have  fewer  runs  and  some  of  them  even  incorporate  variations  in  them.  They  try  to  keep 
main  effects  separate  from  the  interactions  and  find  the  influences  of  each  of  these  on  the  experiment  by 
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solving  them  individually.  They  also  try  to  maintain  same  number  of  runs  at  all  levels  of  each  faetors. 
From  some  of  Taguchi  methods  to  other  individual  ones  achieve  these  goals  with  ease  where  they  solve 
right  from  14  to  100  factors.  The  following  subsections  discuss  these  methods  in  detail. 

6.1  Latin  Square  Design 

A  Latin  Square  design  is  used  when  there  is  a  need  to  compare  the  treatments  and  to  control  up  to  2 
known  sources  of  variation.  As  the  other  designs  these  were  also  first  used  to  design  agricultural 
experiments  (Figure  20).  The  fertility  trends  were  seen  to  run  up  and  down  and  across  the  field.  In  such  a 
scenario  if  there  were  4  fertilizers  used,  then  the  field  would  be  divided  into  16  smaller  plots  by  running  4 
horizontally  and  4  vertically.  So  when  a  Latin  square  design  is  used  in  sueh  a  scenario,  the  Latin  square 
design  will  allocate  the  four  different  types  of  fertilizers  in  these  plots  with  each  type  oeeurring  only  once 
in  a  row  and  column. 


Figure  20.  Example  of  a  Latin  square  design  applied  in  the  field  of  agriculture. 


In  the  early  70’s  researchers  such  as  Finney  [28][29],  Federer  [30],  Freeman  [31][32][33]  and 
Addleman  [34]  explored  on  the  concepts  of  generalizing  the  Latin  Square  designs  and  called  them  F 
square  designs.  Hedayat  and  Seiden  [5]  studied  this  concept  in  detail.  In  F  square  designs  instead  of  the 
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number  appearing  once  in  a  row  and  a  column,  the  number  appears  same  number  of  times  in  the  rows  and 
columns.  So  these  designs  are  explored  to  develop  an  understanding  of  them. 

In  a  Latin  square  design  the  domain  is  needed  to  be  known  before  hand.  The  factors  of  interest  have 
more  than  two  levels.  These  factors  are  also  known  well  before  hand.  Each  of  these  factors  appears  only 
once  in  each  row  and  column.  One  of  the  most  striking  feature  of  such  a  design  is  that  there  arc  negligible 
or  no  interactions  at  all.  It  is  one  of  the  more  complex  designs.  But  it  provides  good  results  by  making 
sure  that  the  main  effect  of  one  factor  does  not  bias  the  main  effect  of  other  factors.  Unconfoundcd  main 
effects  are  also  derived  in  this  design.  Nuisance  factors  are  used  as  blocking  factors.  The  blocking  is 
carried  out  in  order  to  randomize  the  design.  Lindquist  argues  that  in  a  single  Latin  square  design,  the 
main  effect  is  confounded  with  the  interactions  of  the  other  two  factors  and  also  with  the  triple 
interaction.  He  also  stresses  that  the  residual  of  such  a  design  is  of  ambiguous  nature.  The  error  is  much 
lesser  than  that  in  a  randomized  block  design  (RBD)  because  there  is  a  blocking  factor  used.  The  most 
common  sizes  of  this  design  are  5x5  and  8x8,  where  a  5x5  design  is  shown  in  Tabic  7. 


Table  7.  A  5X5  Latin  square  design. 
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These  designs  have  also  been  used  in  the  field  of  medicine  for  cross  over  trials  (suggested  by 
Armitage  and  Berry  [35]).  This  concept  was  further  studied  by  Clark  O  Neil  et  al.  [36]  where  they 
investigated  10  products  on  each  of  the  patients.  Since  it  needed  to  be  orthogonal,  10  patient  sets  were 
chosen  and  a  complete  analysis  was  carried  out.  The  benefits  of  using  this  design  in  a  cross  over  trial  were 
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discussed  in  the  paper  by  Kramer  and  Glass  [6].  This  design  has  also  been  used  in  the  field  of  psychology, 
though  very  rare.  In  a  paper  by  Gatio  [7],  the  author  discusses  the  inherent  defects  in  the  significance  tests 
of  a  Latin  square  design  when  interactions  occur  under  four  conditions  namely: 

•  One  random  variate  model 

•  Two  random  variates  model 

•  Three  random  variates  model 

•  Four  random  variates  model 

Significance  tests  were  carried  out  on  all  these  conditions  for  all  the  effects  and  their  interactions.  The 
author  concludes  that  before  using  this  design,  the  user  must  get  familiar  with  all  the  scenarios  under 
which  significant  results  arc  produced  by  this  design.  Also  he  says  that  in  a  psychological  study,  mainly 
the  first  two  scenarios  are  prevalent.  Information  technology  is  also  not  very  far  from  the  other  fields  in 
the  use  of  this  design. 

In  the  field  of  computer  science,  this  design  has  been  particularly  used  in  compiler  testing.  Robert 
Mandl  [8]  uses  this  design  to  verify  that  evaluation  of  the  operators  on  the  enumerating  values  in  ADA  is 
correct  even  if  these  values  were  in  ASCII  code.  The  author  illustrates  this  by  using  an  example  from  a 
test.  He  concludes  that  the  design  provides  comparatively  great  results  in  form  of  information  for  the 
amount  of  effort  one  puts  in.  He  even  feels  it  is  quite  a  cost  effective  solution.  The  other  common 
applications  of  these  designs  have  been  in  the  field  of  animal  nutrition,  insecticide  field  trials  and  even 
greenhouse  effects.  There  are  two  variations  of  this  design.  But  a  detailed  analysis  is  out  of  the  scope  of 
the  thesis. 

•  Graeco  Latin  Square  Design  -  It  is  a  kxk  design  just  like  the  Latin  square  design  but  the 
information  gathered  from  this  design  is  more  than  just  a  normal  Latin  square  design.  For  example 
if  a  3x3  design  is  considered  one  can  evaluate  the  main  effects  of  four  3  level  factors  in  just  9  runs. 


39 


CUBRC  |  Large  Factor  DOE 


•  Hyper  Latin  Square  Design  -  It  is  also  a  kxk  design  similar  to  the  Graeeo  Design  but  differs  from 
it  as  it  uses  more  number  of  blocking  faetors.  For  example  if  a  4x4  design  is  considered  one  ean 
evaluate  main  effeets  of  five  4  level  faetors  in  just  1 6  runs. 

6.1.1  Advantages 

•  Several  nuisanee  faetors  are  handled  with  these  designs  either  when  they  need  to  be  treated 
separately  or  they  should  not  be  combined  in  to  one  faetor. 

•  Numbers  of  runs  are  very  small. 

•  At  least  two  sourees  of  variation  are  controlled. 

6.1.2  Disadvantages 

•  It  should  be  a  square  design  as  in  the  number  of  levels  of  eaeh  blocking  variable  must  equal  the 
number  of  levels  of  the  treatment  faetor. 

•  A  major  assumption  made  by  this  design  is  that  there  are  no  interactions  between  the  blocking 
variables  and  between  the  main  variable  and  a  blocking  variable. 

•  The  degree  of  freedom  associated  with  the  error  term  is  relatively  small  for  a  small  design. 

•  If  the  number  faetors  are  more,  the  design  tends  to  get  bigger  and  the  error  term  associated  gets 
lager. 

•  If  there  are  any  missing  values  then  the  design  beeomes  statistically  too  eomplex. 

•  The  following  interactions  cannot  be  evaluated: 

■  Rows  and  eolumns 

■  Rows  and  treatment  faetors 

■  Columns  and  treatment  faetors 
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6.2  Plakett-Burman  Design 

During  the  initial  stage  of  experimentation  when  there  is  minimal  knowledge  of  the  problem  in  hand, 
screening  experiments  are  eondueted  in  order  to  find  the  major  factors  in  few  runs.  Until  1946,  the  most 
common  design  used  to  eonduet  these  screening  experiments  was  the  fractional  factorial  design.  But  R.L. 
Plaekett  and  J.P.  Burman  in  their  famous  paper  “The  Design  of  Optimal  Multifaetorial  Experiments” 
described  a  new  economical  and  efficient  design  for  screening  experiments.  The  design  was  named  after 
them.  In  an  un-replieated  fractional  factorial  design  the  number  of  runs  is  restricted  as  a  power  of  2 
whereas  in  a  Plaekett-Burman  design  the  number  of  runs  is  treated  as  a  multiple  of  4,  hence  making  it 
economical  in  obtaining  the  faetors  in  fewer  runs.  This  design  is  used  only  when  main  effects  are  of 
importance  beeause  the  main  effects  are  highly  confounded  with  2  factor  interactions.  There  is  no 
defining  relation  beeause  interactions  are  not  identically  equal  to  the  main  effects.  They  are  resolution  111 
design  known  as  saturated  main  effect.  For  example  just  12  runs  would  be  needed  for  up  to  1 1  faetors. 


Table  8  .  The  six-factor  Plaekett-Burman  Design  used  in  the  first  conjoint  study. 
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1,  represents  the  1th  contrast  and  lTln  consist  of  interaction  effects 

There  are  2  kinds  of  Plakett-Burman  designs  -  geometrie  and  non  geometric.  It  is  a  geometric  PB 
design  when  the  number  of  runs  ean  be  depieted  as  a  power  of  2  if  not  it  is  non  geometrie.  The  design  is 
distribution  of  equal  number  of  pluses  and  minus  in  a  column.  For  eaeh  of  the  n-1  columns,  the  design 
allows  the  contrasting  of  data  by  taking  the  difference  between  the  averaged  data  opposite  to  these  signs. 
In  the  geometrie  designs,  the  columns  are  orthogonal.  But  in  a  non  geometrie  design,  the  contrast  eolumns 
are  mutually  orthogonal  but  at  the  same  time  they  ean  be  correlated  to  contrast  columns  of  the 
interactions.  Due  to  this  the  major  factors  may  not  even  show  up  on  the  radar  making  the  analysis 
inaccurate.  Even  though  there  is  such  a  complex  alias  structure,  the  design  under  some  cireumstanees 
works  and  also  estimates  the  interactions  simultaneously  whieh  other  designs  are  unable  to  do. 

In  order  to  analyze  the  complexity  of  the  alias  structure  formed  in  this  design,  many  authors  have 
proposed  various  methods.  During  screening  experiment,  based  on  the  assumption  that  only  some  factors 
are  the  cause  of  variation  in  the  experiment  (Box  and  Meyer  [37])  and  that  for  an  interaction  to  be 
significant,  the  corresponding  main  effects  also  should  be  significant,  Hamada  and  Wu  [38]  proposed  one 
such  method  in  whieh  they  contrast  the  main  effects  and  two  faetor  interactions  orthogonally  to  the  ones 
found  by  standard  methods.  But  their  design  is  limited  to  2  faetor  interactions.  Some  studies  by  Hyncn 
[39]  showed  that  due  to  aliasing  in  the  design  unwanted  two  faetor  interactions  would  appear  if  the  major 
factors  were  more  in  number.  Box  and  Meyer  [40]  also  proposed  a  method  which  employed  Bayesian 
methodology  in  order  to  determine  if  the  faetor  is  a  major  faetor  or  not.  In  the  paper  by  Tyssedal  and 
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Samset  [9],  the  authors  try  to  make  the  PB  design  more  robust  design  by  trying  to  overcome  the 
shortcomings.  They  provide  an  alternative  method  for  analyzing  the  design  by  understanding  the  alias 
structure  and  by  finding  consistency  between  the  analysis  and  the  projection  properties  of  the  PB  design 
[10]. 

The  Plackett-Burman  designs  have  seen  light  in  various  fields  of  research  and  practical  applications 
for  screening.  In  the  paper  by  Devos  et  al.  [11]  the  authors  use  this  design  to  calibrate  partial  least  square 
regression  model  which  was  being  used  to  test  six  polyclinic  aromatie  hydrocarbons  and  compare  it  with 
the  ones  calibrated  using  eollinearity.  They  conclude  that  the  results  obtained  by  using  this  design  arc 
optimal.  This  design  can  easily  solve  more  than  27  faetors  in  just  28  runs.  Plackett  Burman  design  has 
also  been  used  in  the  field  of  Biotechnology  to  find  the  effect  of  the  medium  in  xylanase  production  using 
a  12  trial  design  [41]  (Li  et  al.). 

Tyssedal  and  Samset  [9]  use  a  12  run  9  response  non  geometric  PB  design  to  make  an  injection 
moulding  environment,  where  15  to  20  variables  are  considered  while  producing  a  new  plastie 
component.  It  is  seen  to  be  a  more  cost  effective  solution  and  also  eyele  times  are  reduced.  The  design  is 
also  used  to  obtain  significant  parameters  rapidly  and  objectively  in  a  thermal  proeess  which  synthesizes 
compounds. 

Another  application  of  these  designs  has  been  in  agriculture  to  find  the  main  extraction  factor  that 
affeeted  the  yield  and  quality  of  pectins  in  ehieory  root.  The  authors  decided  to  use  a  two  level  design 
with  17  factors  and  20  experiments  because  they  faced  a  problem  as  there  were  too  many  faetors  and  were 
unsure  of  the  settings  which  produced  optimal  results  [13]. 

One  of  the  main  applications  of  these  designs  is  in  the  simulation  experiments.  They  help  in  screening 
out  the  important  faetors  used  in  simulating  the  design.  They  are  used  in  setting  up  and  analyzing  the 
computer  architecture  simulation  experiments.  The  solution  was  proposed  to  use  sueh  a  design  as  the 
authors  observed  that  the  processor  simulation  does  not  follow  any  particular  statistical  method  and  the 
results  are  not  in  any  confidence  levels.  Taking  this  eoneept  further,  Vanderster  et  al.  [15]  use  this  design 
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to  optimally  select  the  parameters  for  a  knapsack  metascheduler  which  provides  a  way  to  systematically 
allocate  policies  on  a  computational  grid. 

A  Plackett  Burman  design  generally  produces  a  saturated  design.  In  order  to  solve  these  saturated 
designs  a  new  method  called  Fixing  Effects  and  Adding  Rows  (FEAR)  has  been  discussed  in  a  paper  by 
Heyden  et.al.  [14]  Here  the  authors  have  described  a  model  by  adding  zero  effects  rows  to  the  model 
matrix  after  which  the  largest  estimated  effect  is  fixed  in  order  to  examine  the  factor  effects  accurately. 
This  method  helps  in  estimating  the  effects  of  the  factors.  With  a  set  of  data  values,  a  comparison  between 
FEAR  and  the  conventional  Multiple  Linear  Regression  method  was  carried  out  and  from  the  results  it 
can  be  seen  that  the  new  proposed  method  performs  better  than  the  conventional  method  of  regression 
because  the  main  effects  that  are  significant  are  estimated  more  accurately  in  FEAR  than  when  compared 
to  Multiple  Linear  Regression.  If  the  PB  design  is  complete  then  the  error  is  very  less  even  if  not  they  arc 
insignificant  and  the  design  produces  satisfactory  results  [14]. 

6.2.1  Advantages 

•  Economical  for  detecting  large  main  effects  (assuming  that  all  the  interactions  arc  negligible  in 
comparison  with  few  important  main  effects). 

•  Due  to  the  confounding  the  negligible  impact  is  averaged  and  information  may  be  obtained  about 
significant  interactions. 

6.2.2  Disadvantages 

•  Due  to  confounding,  the  presence  of  a  large  interaction  may  distort  the  effect  of  an  individual 
factor. 

6.3  Split  Plot  Designs 

Split  Plot  designs  are  one  of  the  most  robust  designs  found.  Though  Taguchi’s  approach  is  also  robust; 
the  size  of  the  experiment  is  large  and  needs  a  large  number  of  trails.  Recognizing  this  drawback,  Box  and 
Jones  [42]  suggested  the  use  of  Split  Plot  designs,  which  save  the  number  of  runs  and  even  provide 
additional  information  Split  plot  designs  are  an  extension  for  randomized  block  designs.  These  designs 
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are  used  in  an  industrial  application  when  there  are  moments  where  various  faetors  (processing  and  set 
up)  and  levels  are  difficult  to  understand  or  the  experiment  is  more  expensive  or  it  is  a  very  laborious  and 
time  consuming  experiment.  To  overcome  these  problems,  the  re  ordering  of  runs  is  earned  out  which 
results  in  a  Split  Plot  Design.  Instead  of  carrying  out  expenments  and  then  reordering  it  into  a  Split  Plot, 
the  design  can  be  used  in  the  first  place.  Many  a  times  this  goes  unnoticed.  In  a  paper  by  Kowalski  and 
Potener  [16],  there  are  guidelines  to  reeognize  a  Split  Plot  diagram.  They  say  that  these  designs  have  three 
main  eharaeteristies: 

1 .  The  levels  of  the  faetors  are  not  random  and  are  reset  after  eaeh  run. 

2.  For  eaeh  faetor  the  size  of  the  experiment  varies. 

3.  The  random  assigning  of  the  treatment  combinations  to  the  experimental  unit  is  not  allowed. 

Though  large  in  size  these  are  cost  effective  design  as  shown  by  Webb  et  al.  [44].  Also  in  the  paper  by 

Bisgaard  [43],  there  is  a  cost  model  for  such  a  design  along  with  examples.  These  illustrations  also  show 
a  reduction  in  the  runs  and  additions  in  the  information  obtained. 

There  are  2  levels  or  as  Goos  and  Jones  [45]  say  strata  in  these  designs.  The  experimental  runs  are 
divided  amongst  these  strata.  The  upper  stratum  is  made  up  of  whole  designs  and  the  lower  stratum 
consists  of  subplot  runs.  The  whole  plots  are  a  group  of  runs  where  the  faetor  which  is  difficult  to  change 
remains  constant.  This  nomenclature  of  the  strata  came  to  existence  as  these  designs  were  like  many  other 
designs  used  in  agriculture  originally.  For  example;  the  use  of  the  fertilizer  or  an  irrigation  method  as  one 
of  the  faetors  that  can  be  applied  to  the  large  sections  of  land  called  whole  plot  came  to  be  known  as  the 
whole  plot  faetor  and  the  faetor  associated  with  variety  of  seed  to  be  ploughed  in  various  sections  of  this 
land  by  splitting  it  into  sub  plots  came  to  be  known  as  the  subplot  faetor.  Sueh  a  design  is  used  only  when 
there  are  many  stages  in  the  experimental  design.  There  can  be  reordering  of  designs  within  these  strata 
which  give  rise  to  split-  split-  plot  diagrams.  The  most  common  example  used  while  understanding  this 
method  is  the  production  of  cheese,  a  ease  study  by  Sehoen  [46].  There  is  not  mueh  literature  to 
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understand  the  details  of  a  three  stratum  experiments.  However,  details  on  how  to  design  and  analyze  two 
level  factorial  and  fractional  factorials  are  provided  by  Bisgaard  [43]. 

There  are  a  few  additional  assumptions  that  an  experiment  needs  to  meet  before  the  use  of  Split  Plot 
Design.  They  arc: 

1.  There  are  two  or  more  independent  variables  such  that  one  is  a  non  repeated  measure  treatment  or 
betwccn-block  treatment  and  there  is  at  least  one  repeated,  or  within-block  treatment. 

2.  The  number  of  combinations  of  treatment  levels  is  greater  than  the  desired  number  of  observations 
within  each  block. 

3.  If  repeated  measurements  are  used  on  factors,  then  each  block  will  consist  of  only  one  factor;  if  not 
there  will  be  more  factors. 

4.  The  sequence  for  administering  the  repeated  measures  levels  in  combination  with  each  level  of  the 
non  repeated  measures  treatment  is  randomized  independently  for  each  block. 

The  Split  Plot  Design  is  one  where  a  factor  is  subjected  to  all  levels  of  some  treatments  but  only  one 
level  of  the  other  treatments.  These  designs  combine  both  the  features  between  the  plots  as  well  as  within 
the  plots.  This  feature  helps  in  saving  the  number  of  runs  and  thus  a  large  amount  of  information  is 
gathered  very  efficiently.  The  first  level  factors  are  randomly  assigned  to  the  whole  units  depending  on 
the  whole  unit  design.  The  second  level  factors  are  assigned  to  the  sub  units  randomly  within  each  whole 
unit  according  to  the  rules  of  Randomized  complete  block  design  (RCBD).  Thus  the  entire  design  is 
randomized.  It  would  not  be  too  harsh  to  say  a  good  understanding  of  the  domain  is  needed  in  order  to  use 
this  design. 


Table  9.  Example  of  a  3x2  Split  Plot  Design. 

A I  A2  A3 


B1 

B2 

B1 

B2 

B1 

B2 
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Table  10.  Randomization  of  the  3x2  design  shown  above. 

A1  A2  A3 


B1 

B1 

B2 

B2 

B2 

B1 

When  an  ANOVA  is  conducted  on  such  a  design,  it  is  seen  to  have  two  error  terms.  The  first  error 
corresponds  to  the  pooled  variation  between  the  factors  within  the  groups  and  the  other  corresponds  to  the 
pooled  interaction  of  the  treatment  with  factors  in  each  block.  This  happens  because  there  arc  two 
separate  randomizations  that  occurred  when  the  experiment  was  run  [16].  According  to  Kowalski  and 
Potener  [16]  in  a  Split  Plot  Design  one  needs  to  be  sure  that  there  is  a  true  replication  in  the  whole  plot 
factor. 

From  the  varied  literature,  the  applications  of  this  design  can  be  seen  in 

1 .  Experiments  in  which  the  each  factor  has  a  need  for  different  number  of  experimental  units. 

2.  Experiments  where  one  factor  needs  to  be  more  sensitive  than  the  other. 

3.  Experiments  where  there  is  a  need  to  introduce  new  factors  unexpectedly. 

6.3.1  Advantages 

•  The  sub  plot’s  treatment  factor  and  interactions  are  tested  to  a  generally  high  sensitivity  than  the 
whole  plot  because  the  variance  in  the  former  is  much  higher  than  in  the  variance  in  the  latter. 

•  Experiments  with  a  large  number  of  w  hole  plots  and  lesser  number  of  sub  plots  ean  be  conducted  in 
a  single  experiment. 

•  Factors  may  be  added  with  minimal  additional  cost. 

•  It  is  a  design  where  whole  units  are  subjected  to  repeated  measuring  and  these  repeated  measures 
are  the  sub  plots. 

6.3.2  Disadvantages 

The  design  is  very  robust  however  there  are  a  few  shortcomings  which  are  listed  below. 
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•  The  existence  of  two  error  terms  causes  complicated  analysis  as  there  are  many  standard  error 
comparisons. 

•  The  sensitivity  of  the  whole  plot  is  poor  as  there  is  high  variance  and  few  replications  associated 
with  it. 

6.4  Central  Composite  Design 

Generally  during  the  Design  of  an  Experiment,  mean  models  are  estimated  by  assuming  that  a  design 
is  homogeneous.  Central  Composite  Designs  are  the  second  design  type  discussed  so  far  which  are  used 
in  screening  experiments.  It  was  proposed  by  Box  and  Wilson  in  1951  [47].  These  designs  are  an 
extension  of  the  factorial  designs.  When  either  the  full  or  fractional  factorial  designs  are  embedded  with 
centre  points  and  axial  points  or  star  points;  a  Central  Composite  Design  is  formed  (Figure  21).  The 
centre  points  are  experimental  runs  in  the  design  whose  values  correspond  to  the  median  values  in  the 
factorial  design  and  the  design  is  usually  replicated  along  this  centre.  The  axial  or  star  points  however  are 
the  points  which  aid  in  the  rotation  of  the  design  by  adding  curvature  to  the  design,  mainly  by  including 
the  upper  and  lower  median  values  of  the  two  factorials. 


Figure  21.  Formation  of  a  central  Composite  design. 


A  second  order  central  composite  design  is  an  alternative  design  to  a  3  level  fractional  factorial  design 
or  a  3k  design.  As  k  increases  the  design  size  is  greatly  reduced  in  a  Central  Composite  Design  (CCD).  As 
a  known  fact  such  a  design  contains  twice  the  number  of  star  points  as  the  number  of  factors  present  in  it. 
Like  in  a  factorial  design  one  can  choose  the  value  for  high  and  low  levels,  in  a  composite  design  the 
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values  are  represented  by  these  star  points.  There  are  different  kinds  of  Central  Composite  Designs  based 
on  the  position  of  the  star  points.  They  are: 

•  Circumscribed  composite  design 

•  Inscribed  composite  design 

•  Face  centered  composite  design 

A  Circumscribed  Composite  Design  requires  5  levels  and  the  star  points  define  a  new  high  and  low 
values.  With  these  values  as  a  limit,  if  a  factorial  design  is  created,  then  it  is  an  inscribed  composite 
design  which  also  requires  a  5  level  factorial  design.  In  a  face  centered  composite  design  there  are  3 
levels  of  each  factor.  The  star  points  are  in  the  centre  of  the  faces  of  the  design.  The  inscribed  and  the 
circumscribed  designs  are  rotatable.  But  a  face  centered  design  is  fixed.  The  largest  amount  of  space  is 
covered  by  circumscribed  designs  and  the  least  by  inscribed  design. 
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Figure  22.  Different  Kinds  of  Central  Composite  Design. 

Many  authors  have  used  this  method  to  solve  up  to  120  factors  and  tested  in  evaluating  the  weapon 
system.  In  the  paper  by  Sanchez  and  Sanchez  [17]  the  drawbacks  of  a  large  full  factorial  design  is 
discussed.  Owing  to  this  discomfort  in  solve  a  highly  fractionizcd  factorial  design  is  proposed  that  solves 
large  designs.  These  highly  fractionated  designs  are  the  central  composite  designs.  The  authors  also  show 
that  their  designs  is  double  the  size  when  the  factors  are  in  the  range  of  30-32  and  53-64  but  they  required 
few  centre  points.  With  the  number  of  factors  considered,  there  is  a  variation  in  the  designs.  Some  work 
has  been  carried  out  in  order  to  recognize  these  variations.  In  a  paper  by  Li  et  al  [12],  the  authors 
compare  the  designs  based  on  the  variances  for  both  rotatable  and  non  rotatable  designs.  The  numbers  of 
factors  considered  are  between  6  and  10  with  the  consideration  to  axial  points.  They  conclude  that  the 
CCD  based  on  a  resolution  5  design  performs  really  well  and  in  fewer  runs. 
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Bjorkman  and  Zeius  [18],  discuss  the  application  of  this  design  in  their  paper.  The  challenges  faced  by 
the  military  in  a  decade  to  come,  with  respect  to  the  testing,  have  been  well  recognized  by  the  authors  in 
this  paper.  Further  the  authors  propose  various  process  capabilities  based  testing  methods  such  as  central 
composite  designs  in  order  to  overcome  these  challenges.  The  authors  justify  the  use  of  this  method  by 
saying  that  this  design  would  solve  large  factors  with  low  variance. 

This  design  also  finds  application  in  the  textile  industry.  In  a  paper  by  Kothari  et.  al.  [19]  the  authors 
use  the  design  in  order  to  understand  the  factors  that  affect  air  jet  texturing  which  forms  neps  which 
reduces  the  quality  of  the  yam.  Similarly,  the  designs  have  been  used  in  the  manufacturing  industry  to 
find  the  optimal  factor  for  the  flux  cored  arc  welding  and  to  optimize  the  design.  Similar  to  the  former 
paper  the  authors  recognize  4  factors  which  are  used  to  optimize  the  design. 

The  Central  Composite  Designs  is  a  response  surface  methodology  and  hence  the  entire  surface  is 
under  the  study.  New  work  has  been  carried  out  by  Hader  and  Park  [48]  on  these  designs  to  make  their 
slopes  rotatable  which  reduce  the  variances  in  these  designs.  This  rotatability  is  achieved  by  adjusting  the 
distances  between  the  axial  points.  Sometimes  in  certain  experiments,  the  factors  cannot  be  changed 
easily.  But  Kowalski  et  al.  [20]  have  worked  towards  developing  a  better  understanding  of  this  problem 
and  providing  a  solution  to  it.  Kowalski  et  al.  [20]  modified  these  designs  to  accommodate  a  split  plot 
structure.  Here  the  authors  modified  the  Central  Composite  Design  which  helps  in  the  estimation  of 
different  models  based  on  their  mean  and  variance  under  a  split  plot  structure  taking  it  a  step  further. 
These  designs  have  also  been  used  to  determine  the  property  and  structure  of  certain  epoxies.  In  the  waste 
water  treatment,  these  designs  have  been  used  to  optimize  the  parameters.  Such  is  the  diversification  of 
the  use  of  these  designs. 

Non-central  composite  designs  are  an  attractive  alternative  to  the  central  composite  designs  when  the 
design  is  asymmetric  by  the  shift  of  interest  after  conducting  an  initial  2  level  design.  In  a  paper  by 
Robert  Mee  [21],  the  author  discusses  this  alternative  and  gives  instances  to  support  his  study. 


CUBRC  |  Large  Factor  DOE 


51 


6.4.1  Advantages 

•  The  designs  can  be  run  sequentially. 

•  They  are  very  efficient  as  they  provide  a  lot  of  information  on  the  variables  in  fewer  runs. 

•  The  experimental  error  is  also  determined  in  very  few  runs. 

•  The  CCD’s  are  very  flexible. 

6.4.2  Disadvantages 

•  It’s  a  resolution  5  design  which  is  higher  than  some  other  designs. 

•  The  surface  plots  are  not  rotatable. 

•  Sometimes  the  interactions  between  some  variables  and  square  terms  are  lost. 

6.5  Taguchi  Methods 

Some  of  the  Taguchi  methods  are  also  used  to  solve  experimental  designs.  These  are  designs 
pioneered  by  Dr.  Genichi  Taguchi,  which  have  helped  in  process  improvement  by  improving  the 
productivity.  This  method  is  a  philosophy  by  itself.  In  itself  it  has  2  main  doctrines: 

•  To  decrease  the  inherent  variation  in  any  process 

•  To  develop  a  strategy  in  order  to  carry  out  the  above  stated  doctrine. 

Some  of  these  strategies  could  be  to  identify  which  of  the  parameter  in  the  process  will  help  in 
improving  the  strategies.  It  could  also  be  done  by  identifying  an  alternative  which  will  yield  the  same  or 
better  results.  By  far  it  is  one  of  the  most  robust  designs  known  in  the  industry.  The  noise  factors  and  cost 
of  failures  are  incorporated  in  the  designs  which  ensure  customer  satisfaction.  These  designs  are  used  for 
optimizing  the  design  of  performance,  quality  and  costs  of  any  equipment.  Taguchi  method  for  designing 
an  experiment  is  used  mainly  in  manufacturing  processes.  Figure  23  below  gives  a  pictorial  representation 
of  the  design. 
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Figure  23.  Taguchi  System  Representation. 


It  is  one  of  the  highly  debated  methodologies  [22].  In  [22]  T.N.  Goh  discusses  the  issues  as  to  why 
some  authors  support  this  method  and  some  oppose.  He  says  the  argument  lies  in  the  technical  merit  of 
this  method.  Some  authors  say  that  this  method  has  no  alternative  that  could  predict  the  improvement  But 
the  believers  say  it  is  easy  to  be  used  even  by  a  person  who  does  not  have  in  depth  idea  about  the 
mathematics  of  the  method.  Extremist  also  argue  that  the  variations  in  the  environment  are  not  considered 
and  the  design  can  “hide”  the  requirements  for  optimization  of  the  response  and  minimization  of  the 
variance  in  the  design  making  it  counterproductive.  But  these  designs  have  been  widely  used  and  the 
results  are  hard  to  ignore  which  makes  it  a  valid  design  for  consideration  in  the  thesis.  Mainly  Taguchi 
designs  can  be  broadly  classified  as  Orthogonal  Arrays  and  Linear  Graphs. 


In  an  orthogonal  array  experiment  the  columns  of  the  independent  variable  are  “orthogonal”  to  each 
other.  They  are  often  used  when  there  are  a  number  of  control  factors  in  the  experiment.  These  designs 
arc  fractionated  factorial  designs.  Orthogonal  arrays  have  to  be  defined  in  terms  of  the  number  of  factors 
considered,  the  levels  of  the  factors  and  the  specific  interactions  of  interest. 


Table  11.  Orthogonal  array  eleven  two  level  factors. 


Experiment 


Column 
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Number 

1 

2 

3 

4 

5 

6 

7 

i 

1 

1 

1 

1 

1 

1 

1 

2 

1 

1 

1 

2 

2 

2 

2 

3 

1 

2 

2 

1 

1 

2 

2 

4 

1 

2 

2 

2 

2 

1 

1 

5 

2 

1 

2 

1 

2 

1 

2 

6 

2 

1 

2 

2 

1 

2 

1 

7 

2 

2 

1 

1 

2 

2 

1 

8 

2 

2 

1 

2 

1 

1 

2 

When  there  is  a  need  to  assign  the  factors  in  an  orthogonal  array,  linear  graphs  are  used.  These  are 
substitutes  for  triangular  table,  also  a  Taguehi  method.  As  mentioned  in  [27]  by  the  author,  the  graphs 
have  nodes  and  lines.  The  numbers  present  beside  the  vertices  and  the  edges  correspond  to  the  columns  in 
the  orthogonal  array.  The  vertex  of  the  graph  shows  the  factor  and  the  edge  shows  the  column  of  the 
interaction  between  the  connected  vertices. 


Figure  24.  L8  linear  graph. 

The  operational  steps  of  a  Taguehi  design  is  depieted  in  the  flowchart  below  (Figure  25) 
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Figure  25.  Operational  steps  of  the  design. 


In  the  paper  by  Antony  et.  al.  [23],  the  authors  have  used  the  Taguchi  methods  in  an  automotive 
industry  to  develop  a  new  coil.  An  experiment  is  designed  with  16  trials  to  study  14  parameters  with  one 
interaction.  The  authors  [23]  follow  the  steps  of  the  Taguchi  method  diligently  and  come  up  with  optimal 
settings  for  the  design  parameters  that  arc  very  important  in  making  this  coil.  In  conclusion  they  talk 
about  the  effect  of  such  designs  in  solv  ing  large  problems  easily  in  industry. 

6.5.1  Advantages 

•  Docs  not  consider  specification  limits  but  the  quality  of  the  system. 

•  Error  is  resubmitted. 

•  Uses  the  noise  factors  in  the  experiment. 
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6.5.2  Disadvantages 

•  Orthogonal  arrays  are  used  without  a  thought. 

•  Interactions  are  not  considered. 

•  There  is  no  modeling  but  analysis  results  in  the  answer. 

6.6  Analysis  of  the  Designs 

6.6.1  ANCOVA 

In  the  former  section  of  this  chapter,  the  different  methods  that  are  used  to  solve  large  factor  problems 
are  learnt.  Generally,  after  conducting  the  experiment,  an  ANOVA  is  conducted  to  find  out  the  variances 
in  the  design.  ANCOVA  is  a  method  that  takes  the  analysis  of  variance  a  step  further.  ANCOVA  can  be 
pictorially  defined  as: 

ANOVA 

+ 

Linear  Regression 


ANCOVA 


Figure  26.  ANCOVA  pictiorial  depection 

From  Figure  26  one  can  know  that  ANCOVA  has  two  components  embedded  in  it.  Along  with  a 
regular  ANOVA.  ANCOVA  includes  linear  regression.  In  order  to  understand  this  method  better, 
covariance  needs  to  be  well  understood.  In  simple  terms  covariance  is  the  degree  through  which  two 
variables  vary  together.  A  covariate  is  the  source  of  independent  variation  that  affects  the  response 
variable  but  was  unknown  in  the  beginning  of  the  experiment.  This  is  helpful  because,  it  helps  in  reducing 
the  unknown  variance  in  the  design  which  aids  in  estimating  the  means  of  groups  more  precisely. 
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Knowing  this,  ANCOVA  can  be  defined  as  the  method  that  compares  mean  values  of  response  variables 
between  groups  when  response  variable  co  varies  with  other  continuous  variables  [24], 

6.6.1. 1  Formulation 

Mathematically  ANCOVA  might  be  represented  as 
yiJ=V  +  ai+/3(xiJ-x)+£y 
Where 

yij=  jth  replieate  of  ith  level  response  variable 
ju  =  mean  value  of  response  variable 
a{=  //j  —  ju=  difference  between  the  means 
(3  =  combined  regression  eoeffieient 

xl}  =  eovariate  of  the  jth  replieate  observation  from  the  ith  level  of  a  faetor 
x  =  mean  value  of  eovariate 

€ij  =  unexplained  error  associated  with  jth  replieate  observation  from  the  ith  level  faetor 

6.6.1. 2  Application  of  ANCOVA 

Since  there  is  a  eovariate  used  in  this  method,  the  residual  variation  is  removed.  The  method  hence  can 
test  whether  certain  factors  have  effect  very  easily.  It  is  known  to  statistically  more  powerful  than  a  one 
way  or  even  a  two  way  ANOVA,  since  it  accounts  to  some  variability  in  the  designs.  Adding  a  eovariate 
to  ANOVA  reduces  the  degrees  of  freedom  of  the  design.  But  it  is  a  dependent  on  there  being  a 
correlation  between  the  eovariate  and  the  response  variable.  Adding  a  eovariate  which  accounts  for  very 
little  variance  in  the  dependent  variable  might  actually  reduce  the  statistical  powers  if  not  the  power  of  the 
design  is  very  high. 

6.6. 1 .3  ANOVA  for  ANCOVA 

An  ANOVA  table  of  ANCOVA  is  represented  as  shown  in  Table  12. 
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Table  12.  ANOVA  tabic  for  ANCOVA 


Source 

Df 

IMS 

F-Ratio 

Factor  A  (Adjusted) 

(P-1) 

^A(adj) 

(P-  1) 

MSr  c  siju  aiajj) 

Residual  (Adjusted) 

P(n-1)-1 

cc 

^ Rcsidua{adj ) 

p(n- 1)-1 

Total  (Adjusted) 

pn-2 

By  using  this  analysis  a  statistical  control  of  the  error  is  obtained  which  is  a  strong  point  of  the  design. 
There  is  no  upper  limit  in  the  number  of  factors  that  can  be  considered  by  this  design.  But  a  drawback  of 
the  design  is  that  assumptions  need  to  be  met  and  there  needs  to  be  correlation  between  the  covariate  and 
the  response  variable. 

6.6.1.4  Assumptions 

The  assumptions  are  as  follows: 

•  Normality 

•  Homogeneity  of  variances 

•  Independence 

•  Linearity 

•  Covariate  values  should  not  be  different  amongst  the  group 

•  Fixed  covariate 

•  Homogenous  slopes  -  These  have  to  be  tested  for  compulsorily 

Thus  the  designed  can  be  summarized  as  an  extension  of  the  ANOVA  where  covariate  is  included 
which  helps  in  increasing  the  statistical  power  of  the  experiment  with  the  only  limitation  being  - 
assumptions  need  to  be  met. 
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7  Classification  and  Recommendation  of  Designs 

Foremost  idea  of  the  thesis  has  been  to  develop  an  understanding  of  the  different  methodologies  that 
are  used  to  address  the  issues  of  large  number  of  factors.  The  mathematics  behind  the  design,  their 
complexity  and  number  of  factors  they  can  solve,  their  advantages  and  disadvantages  and  the  application 
of  these  designs  has  been  discussed.  But  this  has  been  mainly  through  literature  survey.  Taking  a  step 
further,  these  designs  have  been  analyzed  and  classified  based  on  the  general  understanding.  Based  on 
these  classifications  a  recommendation  for  the  use  of  these  designs  has  been  provided.  This 
recommendation  can  be  used  as  an  aid  in  the  selection  process,  thus  serving  as  a  guideline. 

7,1  Classification  based  on  Advantages  and  Disadvantages 

After  a  thorough  literature  survey  of  various  cases,  the  advantages  and  disadvantages  of  each  design  is 
listed  in  the  following  manner  (Table  13): 


Table  13.  Advantages  and  dis-advantages  of  all  designs. 


Design 

Advantages 

Disadvantages 

•  Several  nuisanee  faetors  are 

•  Number  of  treatments  should  be 

handled  with  these  designs 

equal  to  the  number  of  rcplieates. 

(either  when  they  need  to  be 

•  The  experimental  error  is  likely  to 

treated  separately  or  when 

increase  with  the  size  of  the 

they  should  not  be  combined 

square. 

in  to  a  single  factor) 

•  Smaller  squares  have  fewer 

•  Fewer  numbers  of  runs. 

degrees  of  freedom  for 

Latin  square  design 

•  The  variations  ean  be 

experimental  error. 

controlled  in  2  directions. 

•  The  following  interactions  cannot 

be  evaluated: 

•  The  efficiency  ean  be 

o  Rows  and  columns 

increased  when  compared  to 

o  Rows  and  treatments 

RCBD. 

o  Columns  and  treatments. 

•  Economy  of  samples  and 

ready  analysis. 
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Plackett-Burman 

Design 

•  They  give  same  info  as  2(k"p) 
resolution  111  design  but  with 

your  fewer  trials. 

•  It  is  a  sequential 

experimentation  whieh  is 

valuable. 

•  By  this  design  the  variables 

can  be  enhanced. 

•  It  is  a  feasible  design. 

•  It  provides  robustness  in  the 

produet. 

•  It  helps  in  intelligent  decision 

making. 

•  Helps  in  finding  which 

variable  ean  be  used  to 

change  the  system. 

•  It  does  not  have  the  power  of 

2  restrictions,  since  it  is  a  4N 

design  making  it  more 

flexible. 

•  The  amount  of  a  priori  knowledge 

of  the  experiment  is  important 

while  using  this  design. 

•  Aliasing  pattern  for  such  a  design 

is  very  eomplex. 

•  It  is  not  a  great  idea  to  run  large 

experiment  or  to  depend  on 

strategies  that  do  not  have  the 

possibility  of  resolving  complex 

relationships  among  factors  with 

only  few  additional  runs. 

Taguehi  methods 

•  Compress  the  amount  of  data 

required  to  carry  out  the 

experiments. 

•  Give  benefits  of  multiple 

simultaneous  AB  split  tests. 

•  Allows  the  testing  of  a  few 

pages  of  elements  all  at  once. 

•  Requires  far  lesser  data  than  a 

normal  design. 

•  Gives  a  robust  design. 

•  To  achieve  the  objective  the 

number  of  trails  required  to 

be  carried  out  is  very  low. 

•  Though  the  size  of  the  experiment 

is  larger  than  a  normal  design,  it  is 

small  when  compared  to  some 

other  designs. 

•  Requires  in  depth  knowledge  of 

DOE. 

•  In  order  to  understand  the  results, 

high  statistical  knowledge  is 

required. 

•  The  variable  interactions  are  not 

considered  in  this  design. 
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• 

Power  of  the  experiment  is 

• 

It  does  not  yield  results  if  the 

higher  than  the  others.  This  is 

assumptions  are  not  met  at  least 

because  there  is  a  reduction 

approximately. 

in  error  variance. 

• 

The  dependent  and  the 

• 

By  this  method  residual 

independent  variables  should  be 

variation  is  reduced. 

linear  in  parameters.  This  further 

• 

It  is  highly  powerful  when 

leads  to  the  fact  that  there  should 

compared  to  1  or  2  way 

be  correlation  between  the 

ANOVA.  This  is  because  it 

eovariates  and  the  response 

has  a  greater  ability  to  detect 

variable. 

ANCOVA 

and  estimate  the  interactions 

• 

There  is  an  additional  cost  of 

(within  the  group  as  well  as 

introducing  blocking  factors. 

between  the  groups) 

• 

The  blocking  factors  that  arc 

• 

There  is  availability  of 

highly  correlated  with  the 

extensions  to  deal  with 

dependent  variables  become  hard 

measurement  errors  in 

to  find. 

covariates. 

• 

If  the  blocking  factor  is  poorly 

correlated,  there  is  a  loss  of 

power 

• 

This  design  reduces  experimental 

error  by  statistical  methods  rather 

than  experimental  methods. 

• 

The  design  gives  highly 

• 

They  employ  5  levels  for  each 

accurate  and  strong  results 

factor,  which  is  higher  than  some 

because  the  detection  limits 

of  the  other  designs. 

are  lowered. 

• 

At  times  such  designs  are  not  able 

Central  composite 

• 

The  design  provides  equal 

to  determine  the  interactions 

Design 

precision  for  fitted  response 

at  the  points. 

between  the  variables  and  the 

square  terms. 

• 

Such  designs  identify 

• 

The  surface  plots  of  such  designs 

multivariable  interactions. 

are  not  rotatable  making  them 

• 

The  numbers  of  trails 

give  different  answers  at  different 

conducted  to  reach  the 

points. 
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required  conclusions  are 

minimized. 

•  Sueh  a  design  also 

determines  the  different 

factor  levels  that  provide 

optimum  responses. 

•  Also  helps  to  determine  the 

portion  of  the  response  that  is 

insensitive  to  changes  in 

predictor  variables. 

•  They  fit  non  linear  models. 

•  These  designs  can  be  used  to 

analyze  data  of  any  kind. 

Split  Plot  Designs 

•  The  sub  unit  variance  in  sueh 

design  is  far  lesser  than  the 

whole  unit  variance.  Thus  the 

sub  unit  treatment  factor  and 

interaction  arc  generally 

tested  with  much  higher 

sensitivity. 

•  Such  designs  ean  earry  out 

both  the  whole  unit  and  sub 

unit  analysis  in  the  same 

expenment. 

•  They  follow  a  univarite 

design  and  have  repeated 

measures  in  time  (sub  unit) 

carry  out  the  whole  unit 

analysis. 

•  Less  precise  than  a  fully 

randomized  experiment. 

•  Many  designs  have  too  few 

degrees  of  freedom  to  give  good 

estimates  of  the  main  plot 

variation. 

•  Analysis  becomes  more  complex 

in  cases  such  as  missing  data 

value,  existence  of  eovariates  or 

while  carrying  out  regression 

analyses. 

•  There  are  2  kinds  of  errors  and 

hence  there  are  many  standard 

error  comparisons. 

•  High  variance  and  few  replication 

of  the  whole  unit  lead  to  poor 

sensitivity  on  whole  unit  faetors. 

•  Presentations  of  results  are  harder. 
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From  the  list  above  certain  inferences  of  the  designs  are  derived.  It  can  be  seen  that  the  designs  have 
certain  common  features.  The  designs  have  been  grouped  further  based  on  these  features  into  classes. 

When  just  the  advantages  are  considered  the  designs  can  be  broadly  classified  as: 

•  Class  1  A- Economical 

•  Class  2A-  Large  amount  of  information  gathered 

•  Class  3  A-  Number  of  runs/trails  are  low 

•  Class  4A-  Control  of  variations 

•  Class  5A-  Aid  in  decision  making 

•  Class  6A-  Solve  Non  Linear  Models 

Though  these  are  self  explanatory,  a  brief  description  of  them  is  discussed.  As  discussed  earlier,  these 
designs  are  generally  economical.  But  this  is  a  very  subjective  interpretation.  Depending  on  the  objective 
of  the  experiment  and  the  situation  of  use,  some  of  the  designs  might  be  more  feasible  than  others. 
Mostly,  a  design  is  economical  when  its  usage  at  least  achieves  a  breakeven.  Some  of  the  designs  take  an 
additional  stride  and  solve  even  non  linear  models.  Most  of  the  designs  discussed  are  robust  in  a  certain 
way,  some  based  on  their  ability  to  control  variation  in  the  process  and  some  based  on  the  amount  of 
information  gathered  which  is  why  they  become  such  an  important  criteria  for  classification.  If  the  later  is 
achieved  in  the  least  amount  of  runs,  the  design  achieves  its  efficiency.  These  designs  can  be  used  in 
decision  making.  How  useful  they  are  as  an  aid,  becomes  another  selection  criterion.  Based  on  the  above 
criteria,  the  following  classifications  of  the  designs  arc  made  (Tabic  14). 


Table  14.  Classification  based  on  advantages. 


Class  1A 

Latin  Square  Design,  Taguchi  Design,  Central  Composite  Design.  Split  Plots, 

Plackett-Burman  Design 

Class  2A 

Plackett-Burman  Design,  Taguchi  Design,  Split  Plots 

Class  3A 

Latin  Square  Design,  Plackett-Burman  Design,  Central  Composite  Design,  Split 
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Plots 

Class  4A 

Latin  Square  Design,  ANCOVA,  Split  Plots 

Class  5A 

Plackett-Burman  Design,  Taguehi  Design,  Central  Composite  Design. 

Class  6A 

Central  Composite  Design 

In  order  to  get  a  quick  idea  as  to  which  of  these  advantages  is  the  most  lucrative  to  be  considered 
while  designing  an  experiment  a  bar  graph  is  quantified. 

6 

5 

4 

3 

2 

1 

0 

Figure  27.  Number  of  designs  for  under  each  class/advantagcs 

To  give  a  fair  idea  of  the  designs,  it  is  unsettling  if  only  the  positive  are  considered.  Therefore,  the 

disadvantages  of  these  designs  also  need  to  be  known.  Based  on  the  list  of  disadvantages,  the  designs 
were  broadly  classified  as 

•  Class  1 D  -  Assumptions  for  the  design  need  to  be  satisfied 

•  Class  2D  -  Complexity  of  the  design 

•  Class  3D  -  Prior  knowledge  of  design  needed  is  high/Domain  should  be  known 

•  Class  4D  -  Limited  analysis  of  the  interactions 
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•  Class  5D  -  High  cost 

•  Class  6D  -  High  Error 

•  Class  7D  -  Low  degrees  of  freedom 

As  we  already  discussed  while  classifying  advantages  this  is  a  very  individualistic  interpretation.  As 
these  designs  are  large,  they  can  get  very  complex  as  the  factors  increase  making  it  hard  to  understand. 
This  intern  leads  to  higher  cost  of  the  design  and  some  of  them  are  associated  with  a  higher  error  term 
caused  due  to  lower  degrees  of  freedom.  Most  of  the  designs  require  a  good  amount  of  domain 
knowledge;  the  assumptions  needed  to  be  satisfied  are  quite  a  few.  Designs  like  Latin  squares  limit 
themselves  to  the  main  effects  and  interaction  analysis  is  not  carried  out.  Hence  these  act  as  the 
parameters  that  measure  the  flip  side  of  the  designs.  After  classifying  the  disadvantages  into  classes,  the 
designs  are  fit  into  these  classes  in  the  following  manner  (Table  15). 


Table  15.  Classification  based  on  disadvantages. 


Class  ID 

ANCOVA 

Class  2D 

Plackett-Burman  Design,  Split  Plots 

Class  3D 

Plaekett-Burman  Design,  Taguehi  Design,  ANCOVA,  Split  Plots 

Class  4D 

Taguchi  Design,  Central  Composite  Design 

Class  5D 

Plaekett-Burman  Design,  ANCOVA 

Class  6D 

Latin  Square  Design,  ANCOVA,  Split  Plots 

Class  7D 

Latin  Square  Design,  Split  Plots 

A  bar  graph  drawn  (Figure  28)  will  aid  in  quantification  of  these  disadvantages  while  designing  an 
experiment. 
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Class  ID  Class  2D  Class  3D  Class  4D  Class  5D  Class  6D  Class  7D 


■  Number  of  designs 
under  each  class 


Figure  28.  Number  of  designs  under  each  class/disadvantages. 

7.2  Classification  based  on  number  of  factors  and  complexity  of  the  design 

As  already  stated,  the  number  of  factors  a  design  can  solve  is  one  of  the  major  reasons  the  research  has 
been  conducted.  In  order  to  achieve  an  optimistic  result  for  this  study,  the  classification  based  on  the 
number  of  factors  is  carried  out.  This  is  a  nascent  step  towards  prov  iding  some  kind  of  guidelines  in  order 
to  achieve  that  tradeoff  between  the  cost  and  the  information  used  and  got  from  these  designs.  The  larger 
the  design  the  cost  is  higher.  So  considering  the  minimum  number  of  factors  that  can  be  solved  with  the 
least  amount  of  effort  the  designs  were  classified  as 

•  High  (x>  15) 

•  Medium(  10<x<  15) 

•  Low(5<x<10) 

where  x  denotes  the  number  of  factors. 

Having  classified  in  this  manner,  the  designs  can  be  grouped  into  these  classes  and  a  graph  is 
developed. 

•  High  -  Latin  Square  Design,  Split  Plot  Design,  Taguchi  designs,  CCD 
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Medium  -  Plaekett-Burman 


•  Low  -  Fractional  Factorial  Design 

The  Figure  29  below  gives  a  quick  update  on  the  designs  and  aids  in  picking  the  most  suited  design  for 
carrying  out  the  experiment.  The  numbers  of  factors  vary  as  high  medium  and  low  and  the  complexity 
involved  in  solving  the  design  varies  from  low  to  high. 


Many 
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Latin  Square 
Designs 


Split  Plot  design 
Taguchi  Designs 


CCD 


Plaekett-Burman  Design 


Fractional  Factorial  Design 


Complexity  involved  in  solving  the  design 


Figure  29.  Complexity  versus  Number  of  factors. 

This  is  the  core  of  the  thesis  and  thus  a  recommendation  is  given  based  on  the  designs  and  their 
behavior  and  features. 


7.3  Comparison  and  Recommendation  of  DOE  Softwares 

A  statistical  analysis  can  be  earned  out  only  with  a  help  of  a  software  package.  The  easier  and  smaller 
experiments  can  be  conducted  intuitively  and  results  calculated  manually.  But  as  the  size  of  the  design 
increases  the  analysis  becomes  more  complex  and  cannot  be  performed  efficiently  without  the  help  of  the 
software  [25].  The  designs  considered  in  this  thesis  solve  large  number  of  faetors  and  are  relatively 
difficult  if  not  for  these  software  packages.  The  packages  today  solve  and  analyze  the  designs  with  ease. 


CUBRC  |  Classification  and  Recommendation  of  Designs 


67 


It  is  possible  for  them  to  carry  out  these  by  fitting  data  into  inbuilt  mathematical  equations  which  predict 
the  outcomes.  These  software  packages  help  in  coming  up  with  optimal  solutions  which  make  it  easy  for 
engineers  to  carve  up  some  savings  in  the  industry.  A  comparative  table  is  shown  below  (Table  16): 


Table  16.  Comparison  of  DOE  softwares. 


Name  of  the 

software 

Features 

Number  of 

Factors 

Cost 

Design  Ease 

1.  Breakthrough  factors  for  process  or 

product  improvement. 

2.  Helps  to  set  up  and  analyze  general 

factorial,  two-level  factorial,  fractional 

factorial  and  Plackett-Burman  designs. 

3.  Numerical  optimization 

Up  to  31 

factors  in 

fractional 

factorial  and 

Plackett- 

Burman 

designs 

For  1  -  2 

$495.00  [26] 

Design 

Expert 

1.  The  peak  of  performance  with  the 

process  or  formulation. 

2.  Has  features  of  Design  Ease  along  with 

in-depth  analysis  of  process  factors  or 

mixture  components. 

3.  Offers  rotatable  3D  plots  to  help 

visualize  the  response  surface. 

4.  Numerical  optimization  function  present, 

which  finds  the  most  desirable  factor 

settings  for  multiple  responses 

simultaneously. 

Include  up  to 

256  runs  and 

up  to  8  blocks 

for  8 -15 

factors 

For  1  -  2 

$995.00 

ANOVATM 

1.  Utilizes  a  complete  set  of  orthogonal 

Solves 

For  1  it  costs 
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Arrays  all  the  way  to  an  L  108  or 

customize  your  own  arrays. 

2.  Calculations  for  Dynamic  Characteristic. 

3.  Individual  Orthogonal  Array  Files  and  4- 

way  Analysis  of  Orthogonal  Array  or 

Single  Factor. 

4.  Auto  pooling  on  V,  F  and  Rho%. 

orthogonal 

Arrays  up  to  L 

108 

Euro  995 

DOE  Kiss 

1.  An  Excel  Add  In  feature 

2.  Solves  Taguchi,  Plackett  Burman  design, 

Full  and  Fractional  Designs 

3.  Computer  Aided  Design  Selection 

Wizard. 

4.  Custom  Designs 

5.  Surface,  Contour,  and  Interaction  Plots 

It  supports  up 

to  26  factors 

and  1  response 

variable. 

Price  per  copy  is 

$249.00 

Statistical 

sample 

planner 

1 .  Helps  pick  the  sample  size. 

2.  Picks  the  best  strategy  that  suits  the 

experiment. 

Price  per  copy 

$40.00 

ECH1P 

Handles  all  the  aspects  of  a  DOE. 

Price  per  copy 

$1495.00 

Minitab 

1 .  Easy  to  use. 

2.  ANOVA,  Regression  analysis,  Statistical 

analysis,  Reliability  analysis,  multi 

variate  analysis,  design  of  experiment, 

response  surface,  surface  plots  etc. 

Price  per  copy 

$975.00 

SAS 

1.  It  has  all  the  regular  features  of  the 

Varies  from 
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statistical  software  along  with  it  they 

have  additional  features  like  to  solve 

MANOVA  and  Split  Plot  Design 

vendor  to  vendor 

SPSS 

Has  all  features  of  SAS  with  compatibility 

For  one  copy 

with  windows 

$1600.00 

Statistica 

1.  Comprehensive,  user  friendly  interface. 

For  one  copy 

more  than  1 1000  functions  interactive  3D 

$1990.00 

explorer  to  name  a  few 

A  detailed  study  of  STASTIST1CA  and  MINITAB  has  been  done.  Both  these  software  packages  have 
their  own  positive  points  and  can  be  used  to  solve  most  of  the  large  factor  designs.  STAT1STICA  is  more 
robust  software  as  it  involves  all  the  minute  details  involve  theoretically  in  a  design.  But  MINITAB  is 
menu  driven  making  it  user  friendly  software  used  by  most  of  the  designers.  Also  it  is  more  economical 
than  the  former. 

8  Conclusions 

A  formalized,  statistically-rigorous  methodology  has  been  proposed  for  the  evaluation  of  any  Data 
Fusion  process.  It  is  shown  that  the  methodology  requires  the  design  of  a  separate  Data  Fusion  process 
that  specifically  supports  the  T&E  process  by  (a)  providing  a  mathematical  approach  to  the  requirement 
to  associate  fused  state  estimates  computed  by  the  SUT  prototype  to  truth  states,  and  (b)  providing  the 
architecture  and  estimation  processes  for  estimating  the  evaluation  metrics  of  interest.  Additionally,  it  is 
shown  that  the  methodology  also  requires,  due  to  the  underlying  stochastic  nature  of  the  DF  process, 
integration  of  the  methods  of  statistical  experimental  design  and  also,  importantly,  the  associated  methods 
of  analysis  that  employ  ANOVA  techniques  and  other  statistical  analysis  methods.  Proof  of  concept 
experiments  have  been  carried  out  to  show  representative  application  of  the  overall  methods;  these 
experiments  and  further  elaborations  of  the  methodological  aspects  are  described. 
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The  increased  complexity  of  modern-day  fusion-based  tracking  problems  requires  formalization  and 
consistency  in  the  PE  process  for  fusion-developed  estimates.  This  gets  harder  for  Level  2,  3  and  4  fusion 
processes  as  explained  in  Section  2.3.  Wc  conducted  empirical  studies  to  bolster  our  understanding  of  the 
complex  interdependencies  in  performance  results  from  changing  SUT/Scenario/PE  parameters.  This 
work  suggests  that  the  nature  of  the  PE  approach  should  build  upon  our  familiarity  with  the  “Fusion  Tree” 
fusion  process  design  for  various  applications.  This  work  also  shows  that  the  quantitative  effects  of 
changing  PE  process  techniques/parameters  can  significantly  affect  the  MOP  results. 

In  the  second  part  of  the  research  the  need  for  large  factor  experiments  was  dealt  by  learning  in  depth 
about  some  methods  that  solved  this  problem.  Some  of  these  designs  have  real  potential  but  have  not  been 
explored  as  they  are  not  well  known  and  they  need  a  thorough  perception.  The  mechanics  behind  these 
designs  are  hard  and  hence  these  are  not  very  popular.  A  classification  of  these  designs  was  carried  out 
and  recommendations  were  made  after  a  thorough  study.  A  comparative  study  based  on  the  complexity, 
number  of  factors,  the  pros  and  cons  has  led  to  a  subjective  interpretation  of  the  design  which  can  be  used 
as  a  guideline.  Finally  the  software  programs  that  aid  in  solving  these  designs  were  discussed  and  a 
comparison  of  these  was  made. 

This  research  is  a  stepping  stone  in  the  world  of  huge  intellectual  opportunities  to  solve  this  non  trivial 
problem.  These  topics  need  to  be  explored  in  detail  and  more  specific  guidelines  need  to  be  set.  The 
designs  can  also  be  understood  in  a  better  way  when  the  results  could  be  quantified  by  plotting  the 
number  of  runs  needed  by  each  design  against  the  number  of  factors  and  also  by  conducting  a  trend 
analysis  of  the  designs.  Further  an  intensive  study  on  the  most  suited  software  can  be  carried  out  and  a 
manual  developed  which  can  aid  experimenters  to  use  these  designs  and  encourages  the  use  of  the  designs 
and  exploit  their  potential. 
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Appendix 

A.  L2  and  L3  Tier  0  DOE  Charts 

This  section  provides  the  Tier  0  DOE  charts  conducted  in  Section  5.  The  three  factors  SUT  Design 
Gating  Factor,  PE  Gating  Factor  and  PE  Design  at  two  levels  each  are  tested  to  find  which  of  these  factors 
affect  the  MOPs  significantly.  In  Tier  0  we  have  three  sensors  on  2  platforms  and  they  do  not  fuse  any 
data  within  or  across  platform.  Hence  we  have  to  only  analyze  track-to-truth  associations  for  each  of  the 
MOPs.  The  summary  of  the  results  is  shown  in  Table  3.  Here  for  each  MOP  we  have  the  Normal 
Probability  plot  and  Pareto  chart  which  summarizes  the  significant  factors.  Then  for  the  significant  factors 
we  plot  the  main  effects  plot  which  tells  us  how  the  change  in  factor  affects  the  MOP.  For  the  significant 
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interactions  we  plot  the  interaction  plot  which  shows  the  effect  of  change  in  factor  level  combination  on 
MOP. 


After  taking  a  look  at  the  summary  (Table  3),  wc  can  say  that  SUT  Design  Gating  Factor  is 
comparatively  more  significant  than  PE  Gating  Factor  and  PE  Design.  SUT  Design  Gating  Factor  appears 
to  be  a  significant  factor  in  nearly  all  the  Tier  0  DOE  runs.  So  at  Tier  0  we  must  be  sensitive  towards 
selection  of  SUT  Design  Gating  Factor. 


Radar  Track  1  to  Truth  Aspect  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  K  Aspect  Alpha  -  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  Aspect  Alpha  -  .10) 


Standardised  effect 


Main  Effects  Plot  (data  means)  for  Aspect 
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Radar  Track  1  to  Truth  ESM  Consistency: 
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Normal  Probability  Plot  of  the  Standardized  Effects 

(response  ks  EM,  Alpha  -  .10) 


FSfctTVp<> 

■  Nets 
I  SqrJ 


Main  Effects  Plot  (data  means)  for  ESM 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  ESM,  A*ha  -  .10) 

1.666 


Radar  Track  1  to  Truth  TTLE  Consistency: 


Radar  Track  1  to  Truth  Threat  Consistency: 
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Radar  Track  2  to  Truth  Aspect  Consistency: 
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Radar  Track  2  to  Truth  ESM  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  ESM_1,  Alpha  -  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  a  ESM_1,  Alpha  «  .10) 


Main  Effects  Plot  (data  means)  for  ESM_1 


Radar  Track  2  to  Truth  TTLE  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  TTLE_1,  Alpha  -  .10) 


A  A 

•  • 

c  c 


Main  Effects  Plot  (data  means)  for  TTLE_1 


Pareto  Chart  of  the  Standardized  Effects 

(response  k  TTLE_1,  Alpha  -  .10) 


1.0  IS 

Standardized  erect 
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Radar  Track  2  to  Truth  Threat  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  Thread,  Alpha  *  .10) 


j  42.4 
42.2 
42.0 
41.8 
41.6 


Pareto  Chart  of  the  Standardized  Effects 

(response  is  Thread,  Alpha  -  10) 

1.666 


1.0  15 

Standardfeed  Effect 


ESM  Track  1  to  Truth  Aspect  Consistency: 
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ESM  Track  1  to  Truth  ESM  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  ESM_2,  Alpha  -  ,10) 


A  A 

1  !• 

C  C 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  ESM_2,  Alpha  -  .10) 
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Main  Effects  Plot  (data  means)  for  ESM_2 
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ESM  Track  1  to  Truth  TTLE  Consistency: 
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ESM  Track  1  to  Truth  Threat  Consistency: 
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ESM  Track  2  to  Truth  Aspect  Consistency 


ESM  Track  2  to  Truth  ESM  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  b  ESM  Jk  Alpha  -  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  ESMJ,  Alpha  -  .10) 


Main  Effects  Plot  (data  means)  for  ESM.  .3 
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ESM  Track  2  to  Truth  TTLE  Consistency 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  TTLE_3,  Alpha  «  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  is  TTl£_3,  Alpha  -  .10) 


ESM  Track  2  to  Truth  Threat  Consistency: 


Probability  Plot  of  the  Standardized  Effects 

(response  Is  Threats,  Alpha  ** .  10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  is  Thr*aL3,  Alpha  -  .10) 


1.666 


Standardized  Effect 


CUBRC  |  References 


86 


1  2 


IRST  Track  1  to  Truth  ESM  Consistency: 


IRST  Track  1  to  Truth  TTLE  Consistency: 
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IRST  Track  1  to  Truth  Threat  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  Threap,  Alpha  -  10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  is  Threats,  Alpha  «  .  10) 


I 


Main  Effects  Plot  (data  means)  for  Threat^ 
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1RST  Track  2  to  Truth  ESM  Consistency: 


IRST  Track  2  to  Truth  TTLE  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  TTLE  ^5,  Alpha  -  .  10) 


Main  Effects  Plot  (data  means)  for  TTLE_S 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  TTLE_5,  Alpha  -  .  10) 

1.666 
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IRST  Track  2  to  Truth  Threat  Consistency: 


B.  L2  and  L3  Tier  1  DOE  Charts 

This  section  provides  the  Tier  1  DOE  charts  conducted  in  Section  5.  The  three  factors  SUT  Design 
Gating  Factor,  PE  Gating  Factor  and  PE  Design  at  two  levels  each  are  tested  to  find  which  of  these  factors 
affect  the  MOPs  significantly  In  Tier  1  we  have  three  sensors  on  2  platforms  and  they  fuse  data  within 
platform  (not  across  platform).  So  we  have  to  analyze  track-to-truth  and  track-to-track  associations  for 
each  of  the  MOPs.  The  summary  of  the  results  is  shown  in  Table  4.  Here  for  each  MOP  we  have  the 
Normal  Probability  plot  and  Pareto  chart  which  summarizes  the  significant  factors.  Then  for  the 
significant  factors  we  plot  the  main  effects  plot  which  tells  us  how  the  change  in  factor  affects  the  MOP. 
For  the  significant  interactions  we  plot  the  interaction  plot  which  shows  the  effect  of  change  in  factor 
level  combination  on  MOP. 
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After  taking  a  look  at  the  summary  Table  4,  we  can  say  that  PE  Design  is  comparatively  more 
significant  than  PE  Gating  Factor  and  SUT  Design  Gating  Factor.  PE  Design  appears  to  be  a  significant 
factor  in  nearly  all  the  Tier  1  DOE  runs.  So  at  Tier  1  we  must  be  sensitive  towards  selection  of  PE  Design 


Track  To  Track  Aspect  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  Aspect,  Alpha  -  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  is  Aspect,  Alpha  •  .10) 


Main  Effects  Plot  (data  means)  for  Aspect 
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Track  To  Track  ESM  Consistency: 


i>i 
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Track  To  Track  TTLE  Consistency: 


Track  To  Track  Threat  Consistency: 
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Track  1  to  Truth  Aspect  Consistency: 
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Track  1  to  Truth  ESM 


Consistency: 


Nonna i  Probability  Plot  of  the  standardized  Effects 

(response  ks  ESM_1,  Alpha  *  .10) 


HfetTrt* 
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Pareto  Chart  of  the  Standardized  Effects 

(response  is  ESM^i,  Alpha  » .10) 

1  666 
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Standardised  Effect 


Track  1  to  Truth  TTLE  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  ta  TTLEL1,  Alpha  -  .10) 
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Pareto  Chart  of  the  Standardized  Effects 

(response  is  TTIE_1,  Alpha  «  .10) 
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Track  1  to  Truth  Threat  Consistency: 


Normal  Probability  Plot  of  the  Standanflzed  Effects 

(response  fe  ThreaO,  Alpha  »  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  ts  ThraaO.  Alpha  *  .  10) 
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Track  2  to  Truth  Aspect  Consistency: 
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Track  2  to  Truth  ESM  Consistency: 


Track  2  to  Truth  TTLE  Consistency: 
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Main  Effects  Plot  (data  means)  for  TTLE_2 
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Track  2  to  Truth  Threat  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  H  Threats  Alpha  -  .10) 


i  tats 

sells 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  TtreatJ*,  Alpha  «  .10) 
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Main  Effects  Plot  (data  means)  for  Threat_2 


C.  L2  and  L3  Tier  2  DOE  Charts 

This  section  provides  the  Tier  2  DOE  charts  conducted  in  Section  5.  The  three  factors  SUT  Design 
Gating  Factor,  PE  Gating  Factor  and  PE  Design  at  two  levels  each  are  tested  to  find  which  of  these  factors 
affect  the  MOPs  significantly.  In  Tier  2  wc  have  three  sensors  on  2  platforms  and  they  fuse  data  within 
and  across  platforms.  So  we  have  to  analyze  track-to-truth  and  track-to-track  associations  for  each  of  the 
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MOPs.  The  summary  of  the  results  is  shown  in  Table  5.  Here  for  each  MOP  we  have  the  Normal 
Probability  plot  which  summarizes  the  significant  factors.  Then  for  the  significant  factors  we  plot  the 
main  effects  plot  which  tells  us  how  the  change  in  factor  affects  the  MOP.  For  the  significant  interactions 
we  plot  the  interaction  plot  which  shows  the  effect  of  change  in  factor  level  combination  on  MOP. 

After  taking  a  look  at  the  summary  Table  5,  we  can  say  that  PE  Design  is  comparatively  more 
significant  than  PE  Gating  Factor  and  SUT  Design  Gating  Factor.  In  this  case  some  of  the  two  way 
interactions  involving  PE  Design  are  significant  which  suggests  that  fusing  data  across  platforms  reduces 
the  discrepancies  in  the  input  data. 


Track  to  Track  Aspect  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  Aspect  Alpha  *  ,10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  is  Aspect  Alpha  «  .10) 


Main  Effects  Plot  (data  means)  for  Aspect 


Track  to  Track  ESM  Consistency: 
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Track  to  Track  TTLE  Consistency: 
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Track  to  Track  Threat  Consistency: 


Track  1  to  Truth  Aspect  Consistency: 


Probability  Plot  of  the  Standardized  Effects 

{response  is  AspecLl,  Alpha  -  .  10) 
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Pareto  Chart  of  the  Standardized  Effects 

(response  is  Aspect_l,  Alpha  »  .  10) 
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Track  1  to  Truth  ESM  Consistency: 


Track  1  to  Truth  TTLE  Consistency: 
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Track  1  to  Trut 
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Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  Threat.!,  Alpha  -  .10) 


Main  Effects  Plot  (data  means)  for  Threat_l 


ireat  Consistency: 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  Threat.!,  Alpha  -  .10) 


Track  2  to  Truth  Aspect  Consistency: 
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Normal  Probability  Plot  of  the  Standardized  Effects 

(response  b  Aspect..2,  Alpha  »  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  Aspects,  Alpha  -  .10) 
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Track  2  to  Truth  ESM  Consistency: 


Probability  Plot  of  the  Standardized  Effects 

(response  b  ESM_2,  Alpha  -  .10) 


Main  Effects  Plot  (data  means)  for  ESM_2 


Pareto  Chart  of  the  Standardized  Effects 

(response  b  ESM.2,  Alpha  -  .10) 
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Track  2  to  Truth  TTLE  Consistency: 


CUBRC  |  References 


103 


Track  2  to  Truth  Threat  Consistency: 
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D.  L2  and  L3  Inter  Tier  DOE  Charts 


This  section  provides  the  Inter  Tier  DOE  charts  conducted  in  Section  5.  The  four  factors  SUT  Design 
Gating  Factor,  PE  Gating  Factor,  PE  Design  and  Tier  Level  at  two  levels  each  are  tested  to  find  which  of 
these  factors  affect  the  MOPs  significantly.  In  Inter  Tier  we  have  three  sensors  on  2  platforms  and  they 
fuse  data  within  and  across  platforms  (Tier  2)  and  just  within  platform  (Tier  I).  So  we  have  to  analyze 
track-to-truth  and  track-to-track  associations  for  each  of  the  MOPs.  The  summary  of  the  results  is  shown 
in  Table  6.  Here  for  each  MOP  we  have  the  Normal  Probability  plot  which  summarizes  the  significant 
factors.  Then  for  the  significant  factors  we  plot  the  main  effects  plot  which  tells  us  how  the  change  in 
factor  affects  the  MOP.  For  the  significant  interactions  we  plot  the  interaction  plot  which  shows  the  effect 
of  change  in  factor  level  combination  on  MOP. 

After  taking  a  look  at  the  summary  Tabic  6,  we  can  say  that  factor  “Inter  Tier”  is  comparatively  more 
significant  than  PE  Gating  Factor,  PE  Design  and  SUT  Design  Gating  Factor.  In  this  case  the  DOE  table 
suggests  that  fusing  data  across  platforms  reduces  the  discrepancies  in  the  input  data. 


Track  To  Track  Aspect  Consistency: 
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Track  To  Track  ESM  Consistency: 


Track  To  Track  TTLE  consistency: 
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Track  To  Track  Threat  Consistency: 
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Track  1  to  Truth  Aspect  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  b  AspecU,  Alpha  -  .10) 
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Pareto  Chart  of  the  Standardized  Effects 

(response  is  Aspect .,1.  Alpha  -  10) 


Track  1  to  Truth  ESM  Consistency: 


Probability  Plot  of  the  Standardized  Effects 

(response  is  ESM_i,  Alpha  -  10) 
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Track  1  to  Truth  TTLE  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  *  TTLE^l,  Alpha  -  .10) 
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Main  Effects  Plot  (data  means)  for  TTLE_1 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  TTIJE_1,  Alpha  -  .10) 
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Track  1  to  Truth  Threat  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  a  Thread,  Alpha  -  .10) 


Pareto  Chert  of  the  Standardized  Effects 

(response  is  Thread,  Alpha  •  .10) 
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Main  Effects  Plot  (data  means)  for  Threat^  1 
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Track  2  to  Truth  Aspect  Consistency: 


Normal  Probability  Plot  of  the  Standardized  Effects 

(response  is  Aspect  .2,  Alpha  *  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  is  AspecU,  Alpha  «  .10) 


Main  Effects  Plot  (data  means)  for  Aspect_2 
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Track  2  to  Truth  ESM  Consistency: 
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Track  2  to  Truth  TTLE  Consistency: 
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Track  2  to  Truth  Threat  Consistency: 


Probability  Plot  of  the  Standardized  Effects 

(response  is  Thre»t_2,  AJptw  »  .10) 
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Main  Effects  Plot  (data  means)  for  Threat_2 


Pareto  Chart  of  the  Standard  zed 

(response  is  Threat_2,  Aiphs  -  .10) 


E.  L4  Tier  0,  1,  2  and  Inter  Tier  DOE  Charts 


This  section  provides  the  Inter  Tier  DOE  charts  conducted  in  Section  5.  The  four  factors  SUT  Design 
Gating  Factor,  PE  Gating  Factor,  PE  Design  and  Tier  Level  at  two  levels  each  are  tested  to  find  which  of 
these  factors  affect  the  MOPs  significantly.  In  Inter  Tier  we  have  three  sensors  on  2  platforms  and  they 
fuse  data  within  and  across  platforms  (Tier  2)  and  just  within  platform  (Tier  1).  So  wc  have  to  analyze 
track-to-truth  and  track-to-track  associations  for  each  of  the  MOPs.  The  summary  of  the  results  is  shown 
in  Table  6.  Here  for  each  MOP  we  have  the  Normal  Probability  plot  which  summarizes  the  significant 
factors.  Then  for  the  significant  factors  we  plot  the  main  effects  plot  which  tells  us  how  the  change  in 
factor  affects  the  MOP.  For  the  significant  interactions  wc  plot  the  interaction  plot  which  shows  the  effect 
of  change  in  factor  level  combination  on  MOP. 
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After  taking  a  look  at  the  summary  Table  6,  we  can  say  that  factor  “Inter  Tier”  is  comparatively  more 
significant  than  PE  Gating  Factor,  PE  Design  and  SUT  Design  Gating  Factor.  In  this  case  the  DOE  table 


suggests  that  fusing  data  across  platforms  reduces  the  discrepancies  in  the  input  data. 


Tier  0  Radar  1  to  Truth  S/J  ration  Consistency 


Tier  0  Radar  2  to  Truth  S/J  ration  Consistency: 
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Main  Effects  Plot  for  Radar 2_Truth_SJ  .Consistency 
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Tier  0  ESM  1  to  Truth  S/J  Ratio  Consistency: 


Main  Effects  Plot  for  ESMl_Truth_SJ_Conslst*ncy 
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Pareto  Chart  of  the  Standardized  Effects 

(response  is  ESMl_Tiuth_S3_Consist)mcy,  Alpha  -  .10) 


Tier  0  ESM  2  to  Truth  S/J  Ratio  Consistency: 
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Normal  Plot  of  the  Standardized  Effects 

(response  is  ESM2_Trutf\  ,SJ.  Consistency,  Alphe  -  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  ESM2  Truth_SJ_Consfefcency,  Alpha  *  .10) 


Main  Effects  Plot  for  ESM2_Truttt_SJ_Conslstency 
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Tier  0  IRST  1  to  Truth  S/J  ration  Consistency: 


Main  Effects  Plot  for  IRSTl_Tmth_SJ_Con»Ist*ncy 
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Tier  0  IRST  2  to  Truth  S/J  ration  Consistency: 


Normal  Plot  of  the  Standardized  Effects 

(response  is  IRST^TnjttLSJ.Cbnststefxy,  Alpha  -  10) 


Pareto  Chart  of  the  Standardized  Effects 
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Tier  1  Track  To  Track  S  /J  Consistency: 


Normal  Plot  of  the  Standardized  Effects 

(response  Is  TradeTo_Trac<.SJ_Conslstency,  Alpha  »  10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  is  T r»ck_T o_T racK_S J_Cors«tency,  Alpha  »  .10) 
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Tier  1  Track  1  To  Truth  S/J  Consistency: 


Normal  Plot  of  the  S  tender  (fixed  Effects 

(resporee  is  Tr*ckl_To_TnJth_SJ_ConsWBncy(  Alpha  ■  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  Track  1JT o_Truth_S J.  Consistency,  Alpha  -  .10) 
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Tier  1  Track  2  To  Truth  S/J  consistency: 
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Main  Effects  Plot  foe  Track2  JTo_Truth_SJ_Consl stency 

Data  Means 


Tier  2  Track  to  Track  S/J  Consistency: 


Normel  Plot  of  the  Standardized  Effects 

(response  Is  Tr»ck_To_Tr»ck_SJ.Cons8tency,  Alpha  -  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  T rackJT o_T rack^S ^Consistency  Alpha  -  .10) 
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Tier  2  Track  1  To  Truth  S/J  Consistency: 
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Main  Effects  Plot  for  Trackl_To_Trutti_SJ_Con»lstency 
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Tier  2  Track  2  to  Truth  S/J  Consistency: 


Normal  Plot  of  the  Standardized  Effects 


Pareto  Chart  of  the  Standardized  Effects 
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Inter  Tier  Track  to  Track  S/J  Consistency: 


Norrral  Plot  of  the  Standardized  Effects 

(response  Is  Track  to  Track  5  J  consistency.  Alpha  -  ,10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  is  Trade  to  Track  5  J  consistency,  Afcha  -  .10) 
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Inter  Tier  Track  1  to  Truth  S/J  Consistency: 
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Main  Effects  Plot  for  Track  1  to  Truth  SJ  Consistency 
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Inter  Tier  Track  2  to  Truth  S/J  Consistency: 


Normal  Plot  of  the  Standardized  Effects 

(response  Is  Tradd  to  Truth  SJ  Consistency,  Alpha  -  .10) 


Pareto  Chart  of  the  Standardized  Effects 

(response  Is  Track2  to  Truth  SJ  Consistency,  Alpha  «  .10) 


Main  Effects  Plot  for  Track2  to  Truth  SJ  Consistency 
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